subagent-driven-development / writing-plans: acceptance tests written by a different agent from the implementer, oracle rules, and a per-task size cap

The problem: tests that pass without measuring what they name

We run SDD-style execution on two game projects (Unity and Roblox), with a fresh implementer and a task review per task, as the skill describes. Over one milestone, the reviews found at least six tests that could not fail for the reason they named. Each was written by the implementer, together with the code, and each inherited the code's view of the world:

• A made-up input. Two tests called the animation code with a synthetic walked distance, and nothing asked what the shipped presenter passes in. The shipped game never advanced the walk, and the tests passed. The defect had been latent since an earlier milestone, and the task's first review missed it.
• The code under test as its own oracle. A per-clip check compared a baked texture with the mesh built from those same texels, so it could only fail on half-precision rounding. Another test checked a muzzle position against the very function that computes it.
• A declared value instead of the observed one. A colour register passed because it checked the declared colour, while the charge effect drew in the threat orange at nightfall.
• A test that pinned the defect. A test asserted that all 20 frames change during a hitstop, which was the defect (feet skating), written down as the requirement.
• A vacuous pass. A slide check that could pass vacuously, and was never shown failing for a slide.

The second project saw the same class independently: a feature's own suite (50 checks) passed, while an independent 84-view run on the real model found 5 failures, which the synthetic fixtures had missed.

The cost was in the fix tails. Five tasks went through 5, 5, 4, 3 and 2 gates. Two of them took 8 and about 5 hours from their first "done" to their merge, and the Critical and most Important findings in those tails were tests that couldn't fail.

Red-first TDD didn't prevent it. The plan said "Write the tests; run them failing" in every task, and the made-up-input tests were red first, against the made-up input. Red first doesn't make the oracle independent; a different author, writing against the shipped entry point, does.

Proposal 1: an acceptance stage in subagent-driven-development

Per task, before the implementer is dispatched:

1. An acceptance subagent (fresh, sized like a reviewer) reads the task's requirements and the spec sections they cite, and nothing of any implementation. It writes one acceptance test per requirement, runs them on the current head, and commits them red. Its report gives, per test, the requirement, the oracle's source and the red failure message.
2. A short test review: a fresh reviewer reads only those tests against the task text, checking the oracle rules below and each requirement's coverage. It is short, because the tests are small and no code exists yet.
3. The implementer is dispatched from the acceptance commit. It adds its own unit tests freely (TDD as today), but changes no acceptance file except a stub's body. If it thinks a test is wrong, it reports BLOCKED with evidence; the controller rules (ledgered, as today's rulings are), and the acceptance agent amends the test, never the implementer.
4. The task review gains two checks:
   • a guard, which is mechanical: the acceptance commit is an ancestor of HEAD, and git diff --name-status ACCEPT HEAD -- <acceptance dir> shows only stub files, whose diffs change only bodies;
   • one named mutant per acceptance test (e.g. "the lookup ignores the slot"), which the implementer shows turning that test red.

Concretely: a new acceptance-prompt.md template beside implementer-prompt.md, a node in the per-task flowchart before "Dispatch implementer", a paragraph in the implementer template, and the two checks in task-reviewer-prompt.md. A ledger line (Task N: acceptance red at <sha> (k tests)) keeps a resumed controller from redoing the stage.

Proposal 2: the oracle rules (in the acceptance template and the task reviewer's checklist)

An acceptance test:

1. drives the shipped path: the real entry point, level, scene, build or binary. It never calls the code under test with an input it made up;
2. has an oracle independent of the code under test: an imported asset, a reference's measured values, the system's own state, a number the spec fixes, or a frozen copy of the code before the change. (For a "no behaviour change" requirement, the function as it stands at the base is copied, renamed, and never edited; the test holds the new code equal to it. Pure functions only.) A function's own output is never its oracle;
3. observes what the user sees: drawn, emitted, returned to the caller. A declared value is not enough. A "looks unchanged" claim needs a before/after capture diff plus a control (a deliberate change) that the same diff must flag;
4. is red for the stated reason: the failure names the missing behaviour, not a compile error or a missing name. Names that the task's Interfaces block produces get stubs that throw not-implemented;
5. cannot pass vacuously: it asserts that what it sampled is non-empty.

Proposal 3: writing-plans gains an Acceptance block and a size cap

• Each task has an Acceptance block: one line per requirement, with what is observed, through which shipped path, and the oracle's source, using the spec's exact values. The acceptance agent writes the tests from it. This replaces the implementer's "Step 1: write the failing test" for acceptance tests; the implementer's own unit-test steps stay.
• A per-task size cap: about 1,500 lines of code diff or 10 code files, and at most two Review Focus items. A bigger task is split by requirement, each part with its own acceptance tests. Our largest task merged 36 code files and 5,172 lines; its implementer reached 92–93% context during its fix rounds, and the other project's workers reached 85–88%. The numbers come from one C# codebase, so they might be a tunable default rather than a constant.
• Task Right-Sizing already says to split "where a reviewer could meaningfully reject one task while approving its neighbor". The cap adds an upper bound to that rule.

Cost

One extra reviewer-sized agent and one short review per task, plus a filtered test run. Against that, a fix tail of 5–8 hours on a large task. If the stage removes one fix round per task, it pays for itself.

Open questions

• Default or opt-in? We'd suggest the default for any task whose requirements a test can observe, with the plan saying so for a task that has none (docs only, a look judged by eye).
• Inline execution (executing-plans) with no subagent tool: the fallback we use is to write the acceptance tests from the Acceptance block before any of the task's code, commit them red, never edit them, and ledger that their author wasn't independent.
• The cap's numbers. Should they be a plan-level setting?

How we'll measure it

Over our next milestone, we'll count: review findings of the "test that couldn't fail" class (target: at most one, against at least six); gates per task (target: a median of 2 or fewer, against about 4); and the time from a task's first "done" to its merge (target: half or less). We're happy to report back, and to open a PR with the templates if the direction is welcome.