#120

triage: unparseable model output is swallowed, dropping a whole batch's triage while reporting success

Openthejesh23opened 3mo ago

What happened

triageBatch in packages/processor/src/triage.ts parses the model's verdict array with a bare JSON.parse wrapped in an empty catch:

const jsonMatch = resultText.match(/```json\s*([\s\S]*?)```/);
const jsonStr = jsonMatch ? jsonMatch[1].trim() : resultText.trim();
let verdicts: TriageVerdict[] = [];
try {
  verdicts = JSON.parse(jsonStr);
} catch {}

On any parse error verdicts stays [], the apply loop runs zero times, and the batch still emits a success message (Batch N/M: 0 triaged). A common trailing-comma or truncated array therefore silently loses the triage for the entire (up to 30-finding) batch — indistinguishable from a clean "nothing to triage" run. There's also no Array.isArray check, so a JSON object throws "not iterable" and is handled inconsistently.

The investigate/revalidate paths were specifically hardened against exactly this: agents/shared.ts routes through jsonrepair and writeParseFailureDebug and fails loud, with a comment noting "a malformed response is otherwise indistinguishable from a clean 'found nothing' run." Triage does neither.

Reproduction

(Found by code audit; regression tests are included in the linked PR.)

  1. Run deepsec triage where the model returns a verdict array with a trailing comma (e.g. [{"title":"X","priority":"P0",...,}]) — a routine LLM formatting slip.
  2. JSON.parse throws, the catch swallows it, and the batch reports "0 triaged" as success. All findings in the batch stay un-triaged with no error.
  3. jsonrepair (already a dependency, used by the investigate path) would have recovered it.

Expected vs actual

  • Expected: minor malformations are recovered via the tolerant parser; genuinely unparseable output fails the batch loudly (and is dumped for debugging), rather than being reported as a clean zero-result run.
  • Actual: any parse error silently drops the batch's triage and reports success.

Environment

  • Found against main (commit around 97ebd04); the code path is present in the current triage.ts.
  • Backend: claude-agent-sdk (the triage path).

Proposed fix

Route triage through the existing parseAgentJsonArray (jsonrepair-backed) + writeParseFailureDebug path used by the other agents, rethrowing on unrecoverable output so the batch is marked failed. PR to follow.

Activity

0

No activity yet.