#118

triage: verdicts correlated to findings by non-unique title, mis-attributing and dropping triage

Openthejesh23opened 3mo ago

What happened

triageBatch in packages/processor/src/triage.ts maps each model verdict back to the finding it judged by title:

for (const verdict of verdicts) {
  const item = batch.find((b) => b.finding.title === verdict.title);
  if (!item) continue;
  ...
}

Finding titles are free-text and not unique — generic ones ("Missing authorization check", "SQL injection", "Hardcoded secret") recur across the up-to-30 findings in a single batch. batch.find returns the first title match, so when two findings share a title:

  • both verdicts are applied to the first finding (the earlier verdict is overwritten by the later one), and
  • the second same-titled finding is never triaged (its triage stays unset), and
  • the p0/p1/p2/skip tallies are mis-attributed.

The batch is rendered to the model as a numbered list (### ${idx + 1}. ${title}), so a stable 1-based index already exists in the prompt — it's just never sent as a field nor used to correlate the reply.

Reproduction

(Found by code audit; a regression test is included in the linked PR.)

  1. Have two MEDIUM findings with the same title (e.g. "Missing authorization check") in different files, both landing in one triage batch.
  2. Run deepsec triage. The model returns two distinct verdicts (say P0 and skip), one per finding.
  3. Both verdicts resolve to batch[0]: the first finding ends up with the second verdict, and the second finding is left un-triaged.

Expected vs actual

  • Expected: each verdict is applied to the exact finding it judged; every finding in the batch is triaged once.
  • Actual: same-titled findings collapse onto the first match; later ones are silently skipped and counts are wrong.

Environment

  • Found against main (commit around 97ebd04); the code path is present in the current triage.ts.
  • Backend: claude-agent-sdk (the triage path).

Proposed fix

Emit the 1-based index shown in the prompt as an id field and correlate the reply on it, falling back to title only when the model omits id, with a guard so a duplicate id/title in the reply can't assign one finding twice. PR to follow.

Activity

0

No activity yet.