Six test-discipline gaps from heavy real-world use
Context: several months of daily superpowers use across multiple projects (TDD, verification-before-completion). Six recurring failure shapes are not covered by the current skill text. Each is independent; happy to split into separate issues if that's easier to triage. Duplicate search done — nearest existing issues are #2146 (RED for the wrong reason), #2092 (arrange step silently fails), #2285 (expected values asserted, not produced), #2286 (evidence provenance) — none carries these.
1. test-driven-development — a mutation that survives with the break verified applied indicts the ASSERTION. When a mutation is confirmed applied (cmp vs backup) but the suite stays green, the current guidance has no branch for it: the finding is about the assertion's discriminating power. The test passed in both states because it asserted on something the feature doesn't change (e.g. a group's item, when a folded group already hides its items). Rule: find the state difference the feature actually makes (a header vanishing, a count changing) and assert on THAT, then re-run the mutation. The first mutation attempt is part of the test's design, not a chore.
2. test-driven-development — write literals in an ESCAPE-PROOF form when they must survive composition through more than one tool layer. An XML-escaped entity written through a quoted heredoc → a python script → an Edit call was silently transformed at each layer (bare quote, then a 5-codepoint char literal, then simply absent); every intermediate artifact looked plausible. Rule: rust/JS unicode escapes, chr() composition, or a single-layer write; and the test's expectation should use the SAME escape-proof form as the code, so both sides carry identical transcription risk and fail together.
3. verification-before-completion — editor diagnostics are cached views; the CLI gate is the authority. tsserver/rust-analyzer update asynchronously and reported the same fixed error five times; twice the right move was NOTHING (the CLI gate was green). Rule: treat tsc --noEmit / cargo build / the project's own gate as the single authoritative verification artifact; editor diagnostics are advisory. When they disagree, the CLI wins. Any edit motivated by a diagnostic gets a CLI re-run AFTER the edit — a fix driven by a stale view shows up as a new CLI failure.
4. test-driven-development — an e2e spec that reloads the page resets its own in-memory mock. A spec drove real UI against an in-memory mock backend, then page.goto()-ed to another route to assert the result. The reload re-imported the mock module and its top-level reset restored the fixture — the assertion silently ran against seeded data, looking like a product bug in a spec that destroyed its own evidence. Rule: route transitions that carry accumulated state happen IN-APP (click the nav link / router push), never via a full page load; where unavoidable, re-seed explicitly. Diagnostic: count the rows first — one probe separates "the app did not record it" from "the fixture forgot it".
5. verification-before-completion — read the runner's VERDICT, never the last line. Playwright/vitest print 2 failed … 30 passed with the pass line LAST. Reading the tail — or grepping only for passed — reports success while the failure line scrolls past unseen; the trap fires exactly on mostly-green runs, the ones one skims. Rule: capture the run and check BOTH failed and passed (or use a reporter that fails the process on any failure); quote counts including failures; never trust a human-read tail for gate decisions.
6. test-driven-development — a test seeding through a production path must re-derive the setup's CURRENT invariants. Two failures, both from dated premises about setup helpers: a "migrations-only database" pattern that no longer leaves the table empty (a since-shipped fix changed it), and a seeder that stores a generated uuid as id with the natural key in external_id — the FK correctly rejected the recorded history key. Rule: derive the setup's current invariants from the code (a one-line probe — SELECT the id, count the rows — before the assertion), and INJECT faults explicitly with a fault-fixture label rather than assuming an old state persists. Sibling of #2285/#2092: setup code is code, and it evolves.
All six were caught in real runs, not reviews; each cost at least one diagnose-fix-retry cycle before the rule was written down. Willing to break these out per-skill or draft the skill-text changes directly if that helps.
For item 2, I would retain escape-resistant writing but qualify the suggestion that the test expectation should use the same form. A shared transcription error can make incorrect code and an incorrect expectation agree; item 1's behavior-level check helps here too.
I ran this small synthetic control on Python 3.12.14 / Windows. It is not a reproduction of the XML incident:
Both bad strings contained code point 8 around
cat. The copied-string equality stayed green, while the independent standalone/embedded-word checks distinguished the two patterns. Python's raw-string documentation explains the literal/regex escaping boundary.A possible wording tweak: use escape-resistant forms when writing both implementation and tests; check the intended behavior or independently specified bytes/code points in the written artifact, beyond equality with a copied literal.
AI-assisted technical note, scoped to this synthetic control.