Report agent never runs set-report-meta with custom models.json provider (DeepSeek) — finalization-report-meta-malformed

Open

ruizdiazeveropened 13h ago

Summary

With a custom models.json provider (DeepSeek via openai-completions), the report agent never calls set-report-meta — it writes .shannon/deliverables/comprehensive_security_assessment_report.md directly instead. The finalizer then rejects the collector's report.json (empty executive_summary) with ReportFinalizationIntegrityError / check code finalization-report-meta-malformed, the Temporal workflow fails, and the CLI exits 1 — even though every other phase (pre-recon, recon, 5 vulnerability agents, exploit agents, reconciliation) completed cleanly.

Deterministic across 3 runs / 2 models:

Run	Model	Mode	report agent behavior
1	`deepseek:deepseek-chat`	pipeline-testing	wrote md, no set-report-meta
2	`deepseek:deepseek-chat`	full exploit-mode scan	wrote md, no set-report-meta
3	`deepseek:deepseek-reasoner`	full exploit-mode scan (reasoner even ran injection-exploit + ssrf-exploit)	wrote md, no set-report-meta

Vulnerability agents do use their harness tools correctly (set_findings_summary, set_strategic_intelligence, submit_exploitation_queue all appear in agent logs), so tool support in general works for this provider — only the report agent's bash-CLI step (set-report-meta via bash) is never attempted (zero calls in agent logs).

Environment

Shannon 3.5.0 via npx @keygraph/shannon@3.5.0 start -u <target> -r <repo> -w <ws> --models-config models.json --follow
models.json: provider deepseek, api: openai-completions, baseUrl: https://api.deepseek.com/v1, models deepseek-chat / deepseek-reasoner
Self-hosted GitHub Actions runner (Docker rootless, compose v2)

Evidence

Kept the worker container alive with --keep-container and read Temporal logs:

[WARN] Activity failed {
  error: ApplicationFailure: Report finalization integrity validation failed.
    type: 'ReportFinalizationIntegrityError',
...
[WARN] Workflow failed {
  error: ApplicationFailure: reporting failed (agent: report)|The scan could not be completed.

The produced report.json:

{
  "report_meta": { "target": "...", "assessment_date": "...", "scope": "...", "executive_summary": "", "exploit": true, "model": "deepseek-chat" },
  "findings": [],
  "reconciliation_failed": []
}

executive_summary is empty because it can only come from set-report-meta, which the agent never invoked (verified across all agent logs).

Expected

The report agent should run set-report-meta --target ... --assessment-date ... --scope ... --executive-summary ... once, as its prompt instructs. Either the command is unavailable/failing silently in the worker container for custom-provider runs, or the prompt section doesn't reach the agent in this configuration — happy to dig further with --keep-container output on request.

Workaround we ship meanwhile

Our CI resumes with the same workspace and stages .shannon/deliverables/ as a job artifact regardless of the finalization verdict, so assessments are still delivered.

Run against our own dev environment (our code, our infra); no third-party targets were touched.

Activity

No activity yet.