Eval provenance: source tags and set hashes (RAI-261, RAI-262)

Draft

tobi-techywants to merge intofrom

Draft. Closes the provenance gaps left on RAI-261 and RAI-262.

"source": "synthetic" on every label line in eval/proof/screens.jsonl (40 payment screenshots) and eval/credit/shelves.jsonl (31 shelf photos). Only that field is added; every other byte of each line is unchanged.
scripts/make_proof_fixtures.py and scripts/make_shelf_fixtures.py write the tag, so a rerun reproduces it.
README, payment-screenshot eval section: a set SHA-256 per eval set (the hash of its label file, which carries each image's own SHA-256, so it pins labels and images).
- screens: 566666af77baed1311cc9881e6523eb227aa4ada84ad8d45d071c9f917c1f4e1
- shelves: f059df42437f28e6afbe2db8e38a37dae2e3f05f61731ebc64d2866f32213f25
tests/test_eval_provenance.py: every label has a source, every image matches its label's SHA-256, shelf source agrees with synthetic, and each label file matches its README hash.

Offline, no model calls. Still open on the issues: linking each screenshot to a seed-world payment row, a held-out screenshot split, Tobi's spot checks.

Activity

Eval provenance (RAI-261, RAI-262): source: synthetic on every payment-screenshot and shelf-photo label, the generators write it, and each set's SHA-256 is recorded in the README and checked by a test

f183b15Oct 30

coderabbitai[bot]1d ago

IMPORTANT

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:
reviews:
  auto_review:
    drafts: true

Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

This pull request is a draft

Merging is unavailable until it is marked ready for review.