#2470

Parser upgrades skip unchanged Codex rollouts previously indexed as raw JSON

Openjdoradoopened 1mo ago

Reproduction

A parser update alone does not repair Codex sessions already indexed through raw JSON fallback if the source file is unchanged.

  1. With the published 3.9.0 Docker image, mine a synthetic conversation generated by Codex 0.153.4. The one-exchange native rollout becomes 116 raw JSON/text chunks.
  2. Preserve the indexed palace and the source file's path, content, and mtime.
  3. Run conversation mine with current develop plus the current-event parsing/incremental changes from #2408 (and the verbatim-default change from #87).
  4. The unchanged source is skipped as already filed. No reparsing occurs, so the raw import remains.

In the three-file reproduction, the upgraded mine reported Files processed: 0, Files skipped (already filed): 3, Drawers filed: 0.

As a separate control, a copied source tree with newer mtimes did re-mine with the same fixed parser and filed the intended seven exchanges rather than the previous 122 total chunks. Merely copying/touching user sources is not an adequate migration contract.

Proposed recovery contract

Consider a per-format normalization revision, or an explicit scoped re-normalize command that bypasses unchanged-source/sentinel skips for selected Codex files. It should use the existing crash-safe refile path, replace only drawers derived from those sources, preserve unrelated data, and be idempotent. A repaired source should not be re-embedded on every subsequent run.

The guard in #2469 prevents new unsupported-rollout fallback, but intentionally does not bump the global normalization version or silently rewrite existing archives. #2233 / #2408 handle current events. Please document how already-filed raw sessions become eligible for recovery alongside the parser fix; the ordinary mine backfill command can otherwise report success while skipping them.

Acceptance: same source bytes/path/mtime, old raw import -> one clean exchange after explicit scoped repair -> zero additional work on the next ordinary mine; interrupted repair must not lose the old content. Unknown future schemas must remain retryable rather than being marked successfully filed.

Only synthetic conversations were used; no private rollout is attached.

Activity

0

No activity yet.