#2553

brainstorming / writing-plans: the spec's claims about today's code are never checked against the code, and the self-review passes clean

Opengugaibarros-cpuopened 11d ago

Skills: brainstorming (Spec Self-Review), writing-plans (Self-Review). Checked against 6.4.2 and 7.0.0.

  • I searched existing issues and PRs and this has not been proposed before (adjacent: #2441 is about code citations for claims in user-facing text; #2432 is about adversarial probes for security boundaries; neither asks for a factual check of the spec's claims about current behaviour).

What problem does this solve?

A spec written from exploration reports (typically read-only subagents citing file:line confidently) is reviewed by the Spec Self-Review: placeholders, internal consistency, scope, ambiguity. That review reads the document against itself, so it cannot see a wrong claim about how the code behaves today.

In one project, two specs were written this way. The self-review passed clean both times. An independent check of every claim against the code then found:

  • Spec 1: 3 claims WRONG and 7 PARTIAL, plus design holes no internal check could see, one of them only visible by querying the real data (the new rule would have broken 216 of 241 real artifacts).
  • Spec 2: 1 wrong line reference (it would have extracted the wrong block), 5 regressions against a "nothing else changes" promise, and 9 holes (for example a duplicated socket with the same client id that would have left a watcher deaf).

The final whole-branch review also produced a finding that the code refuted: the reviewer had not read the code that made it false. A reviewer's finding needs a reproduce-or-refute step before it enters the fix pass.

Proposed solution

  1. Add a fifth Spec Self-Review item. When the spec states current behaviour ("today X does Y"), especially from a subagent report, dispatch a read-only check that returns CONFIRMED / WRONG / PARTIAL per claim with evidence, and asks for regressions against the behaviour the spec promises not to change. Fix before the user gate.
  2. The same check in writing-plans Self-Review for steps that depend on facts about existing code.
  3. A reviewer's finding enters the fix pass only after the executor reproduces it (a failing test) or refutes it in the code.

Principle: a self-review that reads the document against itself cannot find defects whose evidence lives in the code.

Activity

0

No activity yet.