Skills: brainstorming (Spec Self-Review), writing-plans (Self-Review). Checked against 6.4.2 and 7.0.0.
- I searched existing issues and PRs and this has not been proposed before (adjacent: #2441 is about code citations for claims in user-facing text; #2432 is about adversarial probes for security boundaries; neither asks for a factual check of the spec's claims about current behaviour).
What problem does this solve?
A spec written from exploration reports (typically read-only subagents citing file:line confidently) is reviewed by the Spec Self-Review: placeholders, internal consistency, scope, ambiguity. That review reads the document against itself, so it cannot see a wrong claim about how the code behaves today.
In one project, two specs were written this way. The self-review passed clean both times. An independent check of every claim against the code then found:
- Spec 1: 3 claims WRONG and 7 PARTIAL, plus design holes no internal check could see, one of them only visible by querying the real data (the new rule would have broken 216 of 241 real artifacts).
- Spec 2: 1 wrong line reference (it would have extracted the wrong block), 5 regressions against a "nothing else changes" promise, and 9 holes (for example a duplicated socket with the same client id that would have left a watcher deaf).
The final whole-branch review also produced a finding that the code refuted: the reviewer had not read the code that made it false. A reviewer's finding needs a reproduce-or-refute step before it enters the fix pass.
Proposed solution
- Add a fifth Spec Self-Review item. When the spec states current behaviour ("today X does Y"), especially from a subagent report, dispatch a read-only check that returns CONFIRMED / WRONG / PARTIAL per claim with evidence, and asks for regressions against the behaviour the spec promises not to change. Fix before the user gate.
- The same check in
writing-plansSelf-Review for steps that depend on facts about existing code. - A reviewer's finding enters the fix pass only after the executor reproduces it (a failing test) or refutes it in the code.
Principle: a self-review that reads the document against itself cannot find defects whose evidence lives in the code.
Activity
0No activity yet.