#2557

test-driven-development: two blind spots where the test is born from the threat the author already imagined (a stub that pins the predicate; a lock that guards one entry)

Opengugaibarros-cpuopened 11d ago

Skill: test-driven-development (the failing-test-first and watch-it-fail parts). Checked against 6.4.2 and 7.0.0.

  • I searched existing issues and PRs and this has not been proposed before (adjacent: #2092 is about an arrange step that silently never creates the condition; #2494 is about assertions inside plans. Both are about a vacuous test; these two are about a test written from a threat the author already imagined).

What problem does this solve?

1. A stub that pins the condition under repair. A bug fix came with green tests. They used a fixture helper that stubbed isEnabled() to always return true, inherited from older tests. The bug was precisely that the feature was DISABLED after a certain event. The test exercised a state that never happens in production and passed; a re-reviewer who read the helper caught it. Without the stub, the tests reproduced the bug and validated the fix.

2. A busy flag that guards one entry, not the state. A "creating" flag was added to stop a double click from creating two paid resources, and tested against that threat. It guarded only create; use, search, release and status() touch the same state and ignored it. The author's test and self-review asked "does a double click create two?", never "who else writes this state while the operation runs?". A fresh reviewer asked the second question and found a paid resource that could be lost.

Proposed solution

  1. Before trusting a regression test, list the stubs and overrides the fixture applies and check whether any pins the condition the bug depends on. If so, drop it for this test and use the real state. "Watch it fail" catches this only if the RED happens with the exact fixture that stays in the final test.
  2. When code introduces an in-progress flag or lock, list every public method that reads or writes the same state and write one test per method called while the operation is in flight (the fake needs a delay for the operation to exist in the test). For operations that cost money, the final test is "the created resource is never left without a record, whatever the order".

Principle: a test written from the author's imagined threat inherits the author's blind spot.

Activity

1
arittr9d ago

Hi @gugaibarros-cpu — I'm Claude, an AI agent (Claude Opus 5.5, running in Claude Code 2.1.305), posting from Drew (@arittr)'s account at their direction. Drew reviewed the finding below and approved this reply.

We checked both cases against v7.0.1. For the first, verification-before-completion's regression-test pattern (revert the fix, run the test, it must fail, restore) runs the final fixture, so a test whose stub pins the predicate still passes with the fix reverted and that check flags it; the fixture side overlaps #2092. The second case, testing every method that touches the state an in-progress flag protects, isn't covered by the TDD skill today. To weigh it, we need the Environment table the issue template asks for (model, harness, harness version, installed plugins) and a redacted transcript excerpt from the session where those tests were written, showing whether test-driven-development was loaded. Thanks for the report.


If any of the evidence above is wrong, reply here — Drew reads these, and decisions can be revisited.

— Claude Opus 5.5, Claude Code 2.1.305, session 5d9d84c2-4219-41be-8d07-d9ba2ad854be