#404

evals: discrimination checks for --fixture (negative fixtures)

DraftAlemusicawants to merge intofrom

Implements #403.

What

Adds an optional negativeFixtures field to EvalCase: near-miss sources the case's gates MUST reject. In --fixture mode each negative runs through the same local validation path as the golden; the case fails if any negative is not rejected by at least one gate — surfacing cases whose gates are too weak to discriminate a wrong answer.

Three base-case negatives

  • hello-world — drops the trailing \n → caught by the stdout/run comparison.
  • fibonacci — drops the fib(10) == 55 verification term (the run still prints the same line) → caught by requiredSourcePatterns, proving the per-value asserts are load-bearing.
  • scale-multi-command-cli — multiply returns x - y with help and the other commands correct → caught by the multiply 6 7 → 42 runCheck, proving multi-command coverage isn't met by the happy path.

Verified locally

All three goldens still pass and each negative is rejected by its expected gate. A self-test (injecting a valid source as a negative) correctly fails the case with "not rejected by any gate", confirming the check is not a no-op.

Direction (not in this PR)

These negatives are hand-written. A natural extension is to generate near-misses by mutating the golden with a few rules (mutation-testing flavor), so each case proves its own discrimination automatically. Happy to discuss whether that's wanted before building it.

Scope: TS harness + eval methodology only; no compiler changes.

Activity

2
vercel[bot]5mo ago

@alelab is attempting to deploy a commit to the Vercel Labs Team on Vercel.

A member of the Team first needs to authorize it.

This pull request is a draft

Merging is unavailable until it is marked ready for review.