#89
Issues/BaaRai/ARTEX/#89

Benchmark: diversify public ground truth beyond Juice Shop (XBOW set is saturated; Cybench/CyberGym candidates)

OpenBaaRaiopened 1mo ago

Context

The benchmark (#59) validates against Juice Shop only: one ground truth couples the scorecard to one application's vulnerability classes, and agent projects increasingly publish scores on standardized public suites, so an external anchor makes our numbers comparable.

Fact-check on the obvious candidate before filing: the XBOW validation-benchmarks set (104 dockerized CTF-style web tasks, Apache-2.0, per-task flag ground truth, self-hostable via compose) is officially deprecated: as of mid-2026 the set is saturated (standard models score near 100%, repo kept for historical purposes only). So XBOW works as a regression anchor, not as a headline comparability number, and contamination means any saturated-benchmark score overstates skill.

Proposal

Two-tier, scripts-only, reusing the existing judge and falsification battery unchanged:

  1. Saturated anchor suite (XBOW subset). Pick roughly 10 tasks spanning the difficulty levels, self-hosted via their compose files with our own injected flags. Purpose: cheap regression detection. Expected score is near 100%, so the value is in alerting on drops, and the suite is fast and deterministic. Never cite it as a capability number.
  2. Headline ground truth: one unsaturated public suite. Candidates:
    • Cybench: 40 professional CTF tasks, still the standard agent-security benchmark, web plus crypto/forensics/pwn.
    • CyberGym: real-world vulnerability reproduction (OSS-Fuzz based) with sandboxed PoC validation; dual-use, since it is also the harness the P0 validation-loop backlog item needs. Per-suite Wilson intervals as in #59; no mixing of tiers in one number.

Scope

  • scripts/benchmark/ additions plus a compose file alongside docker-compose.bench.yml; no server changes, no rebuild.
  • Everything self-hosted in the bench network, no external targets.

Open questions

  1. Which suite carries the headline number: Cybench (breadth, standard) or CyberGym (real-world alignment, P0 synergy)?
  2. Runtime budget: how long can a full benchmark run take in the bench compose before it stops being run regularly?

Activity

1
BaaRai21d ago

Scoping update before implementation (so the plan is on record):

  • Tier 1 (XBOW saturated anchor, ~10 tasks self-hosted via their compose with injected flags) will land as a new corpus adapter in scripts/benchmark reusing the DrostBench deploy/teardown machinery (corpus-type=xbow). Expected score ~100 percent by design; the value is regression alerting only, never a headline number.
  • Tier 2 (unsaturated headline suite: Cybench vs CyberGym) needs a ground-truth decision first - CyberGym doubles as the P0 validation-loop harness, Cybench is the standard comparability anchor. I will bring a one-paragraph tradeoff to this issue before writing that adapter; per-suite Wilson intervals per #59, tiers never mixed in one number.

Both adapters are deliberate, careful work (the DrostBench adapter shipped with four live-run bugfixes); not rushing them at the tail of a big merge wave. This issue stays the tracking ticket.