#217

fix: parse saved transcripts with the definition's parser

OpenRaphaelFakhriwants to merge intofrom

Summary

Fixes #216.

saveResults chose the transcript parser from the experiment's agent ID. A registered custom agent's ID names no parser, so its transcript.json had no events (parseSuccess: false) and result.json carried an empty o11y summary. Token usage and the EVAL.ts transcript context already used the definition's o11yAgentName, so the saved results disagreed with what EVAL.ts saw.

This change:

  • Adds an optional o11yAgentName to SaveResultsOptions and uses it to pick the parser, falling back to the agent ID as before.
  • Passes the o11yAgentName that runExperiment already resolves to saveResults.
  • Keeps the experiment's agent ID as the saved transcript's agent, so output for built-in agents (for example vercel-ai-gateway/claude-code, whose parser name is claude-code) is unchanged.

Testing

A new test in runner.test.ts registers a custom agent whose definition sets o11yAgentName: 'codex', runs an experiment, and checks the saved transcript.json and result.json.

  • Without the fix: the test fails (expected false to be true on parseSuccess).
  • With the fix: the test passes.

npx vitest run src/lib/runner.test.ts src/lib/results.test.ts passes (50 tests). eslint and tsc --noEmit are clean.

A changeset is included.