#190

Codex session-file transcript may not match the schema the o11y parser expects

Opengaojudeopened 2mo ago

Follow-up from #169, filed rather than blocking that merge.

#169 makes the Codex runner fall back to the saved session transcript from ~/.codex/sessions when codex exec --json does not echo the full JSONL to stdout. That is a clear improvement over recording nothing, which is what happened before. The open question is whether the two sources are interchangeable downstream.

They are both JSONL but appear to carry different event schemas:

  • the session file uses turn_context with a payload wrapper, which is what extractObservedModelFromCodexSession reads (payload.model, payload.collaboration_mode.settings.model)
  • packages/agent-eval/src/lib/o11y/parsers/codex.ts switches on item.started / item.completed, plus response.failed and output_text.delta

If that reading is right, a fallback transcript will not crash, there is a default case, but it may parse to near-zero events. The concern is that transcript-derived numbers would then read as 0 rather than absent, which is worse than a null transcript when results are aggregated, because zero looks like data.

I could not settle this without a real Codex run, so it needs either confirmation that the schemas line up, a normalizer on the fallback path, or scoping the fallback so it populates the raw on-disk transcript without feeding the parsed path.

cc @huang-julien

Activity

0

No activity yet.