#217
fix: parse saved transcripts with the definition's parser
Summary
Fixes #216.
saveResults chose the transcript parser from the experiment's agent ID. A registered custom agent's ID names no parser, so its transcript.json had no events (parseSuccess: false) and result.json carried an empty o11y summary. Token usage and the EVAL.ts transcript context already used the definition's o11yAgentName, so the saved results disagreed with what EVAL.ts saw.
This change:
- Adds an optional
o11yAgentNametoSaveResultsOptionsand uses it to pick the parser, falling back to the agent ID as before. - Passes the
o11yAgentNamethatrunExperimentalready resolves tosaveResults. - Keeps the experiment's agent ID as the saved transcript's
agent, so output for built-in agents (for examplevercel-ai-gateway/claude-code, whose parser name isclaude-code) is unchanged.
Testing
A new test in runner.test.ts registers a custom agent whose definition sets o11yAgentName: 'codex', runs an experiment, and checks the saved transcript.json and result.json.
- Without the fix: the test fails (
expected false to be trueonparseSuccess). - With the fix: the test passes.
npx vitest run src/lib/runner.test.ts src/lib/results.test.ts passes (50 tests). eslint and tsc --noEmit are clean.
A changeset is included.