Implements #403.
What
Adds an optional negativeFixtures field to EvalCase: near-miss sources the case's gates MUST reject. In --fixture mode each negative runs through the same local validation path as the golden; the case fails if any negative is not rejected by at least one gate — surfacing cases whose gates are too weak to discriminate a wrong answer.
Three base-case negatives
- hello-world — drops the trailing
\n→ caught by the stdout/run comparison. - fibonacci — drops the
fib(10) == 55verification term (the run still prints the same line) → caught byrequiredSourcePatterns, proving the per-value asserts are load-bearing. - scale-multi-command-cli —
multiplyreturnsx - ywithhelpand the other commands correct → caught by themultiply 6 7 → 42runCheck, proving multi-command coverage isn't met by the happy path.
Verified locally
All three goldens still pass and each negative is rejected by its expected gate. A self-test (injecting a valid source as a negative) correctly fails the case with "not rejected by any gate", confirming the check is not a no-op.
Direction (not in this PR)
These negatives are hand-written. A natural extension is to generate near-misses by mutating the golden with a few rules (mutation-testing flavor), so each case proves its own discrimination automatically. Happy to discuss whether that's wanted before building it.
Scope: TS harness + eval methodology only; no compiler changes.
Activity
2This pull request is a draft
Merging is unavailable until it is marked ready for review.
@alelab is attempting to deploy a commit to the Vercel Labs Team on Vercel.
A member of the Team first needs to authorize it.