Evaluate zero-shot TF-IDF and compression on JevBench

Draft

mrmpswants to merge intofrom

Measures zero-shot TF-IDF/compression hybrids on JevBench's 231 public typed decisions. Every method sees only the current state, instructions, labels, and option descriptions: no labeled training, support examples, learned weights, or calibration.

Method	Accuracy
Character TF-IDF	49.4%
Best observed fixed TF-IDF + Zstd blend	50.6%
Jev 1.13.0, published on the same public IDs	86.6%

The hybrid is the best of 45 fixed blends in an exploratory sweep. Its three-answer gain over character TF-IDF has a paired 95% interval of −3.07 to +5.73 percentage points, so these results do not establish an improvement. Public accuracy is not an official JevBench score.

The harness includes complete prediction/score artifacts, source hashes, option-order checks, and reproduction instructions. The checked-in report covers all 54 methods. Only offline evaluation files change; visual evidence is not applicable, and deployed behavior is unchanged.

Activity

Measure gzip and Zstd text classification

53c6d2fSep 24

coderabbitai[bot]1mo ago

IMPORTANT

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:
reviews:
  auto_review:
    drafts: true

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

Focus compression study on zero-shot JevBench performance

9005d6eSep 24

This pull request is a draft

Merging is unavailable until it is marked ready for review.