#514

Preserved-thinking Anthropic models (Mythos 5.1 etc.): pre-recon fails with 400 "Invalid signature in thinking block" because pi-ai 0.84.x resends thinking blocks after the tool list changes

Opengsomikopened 3d ago

Describe the bug

A scan on anthropic:claude-mythos-5-1 dies in pre-recon after 35 to 40 minutes with a generic AGENT_EXECUTION_FAILED. Reproduced twice on 2026-11-05, on 3.4.0 and on 3.5.0. The real cause in .shannon/deliverables/error.log is a 400 from the Anthropic API:

400 {"type":"error","error":{"type":"invalid_request_error","message":"messages.1.content.0: Invalid `signature` in `thinking` block. The block is bound to a different conversation. Remove the block, or set `thinking.block_binding.prefix_mismatch_behavior` to \"drop_block\". That setting requires the `thinking-binding-controls-2026-08-01` value in the `anthropic-beta` header. The `tools` list differs from the one this block was created with."},"request_id":"req_011CgjtXaBpCBmbDSr6FbjqV"}

What happens: the pre-recon agent finishes its sub-agents, then the tool list of the conversation changes (the set_executive_summary / set_application_intelligence / set_auth_deep_dive tools appear). The next request replays the earlier signed thinking blocks together with the new top-level tools list. Models with preserved thinking (Mythos 5.1, Fable 5.1, Opus 5.5, Sonnet 5.5, Haiku 5.5) bind each thinking signature to the system prompt, the tool list and the history, so Anthropic rejects the request. The error is not retryable, and the whole scan is lost.

The same scan passed pre-recon on 2026-10-15 with the same Shannon version (3.4.0), so the strict check seems to have reached our Anthropic organisation between then and now.

Root cause, as far as I can see: the worker pins @earendil-works/pi-ai at ^0.84.4 (apps/worker/package.json, same on 3.4.0, 3.5.0 and main). That line has no handling for this check. Pi 1.x fixed it in two ways: it sends thinking.block_binding.prefix_mismatch_behavior: "drop_block" with the thinking-binding-controls-2026-08-01 beta for models with compat.supportsMidConvoEffort, and with compat.supportsMidConvoToolChanges it declares later tools as tool_addition blocks (inline-tools-2026-09-15 beta) so the top-level tool list never changes. See earendil-works/pi#10954 (closed 2026-11-02), earendil-works/pi#10324 and earendil-works/pi#9391. The caret range ^0.84.4 cannot pick up 1.x.

Steps to reproduce

  1. Register Mythos 5.1 through --models-config (the file from #496, with compat.forceAdaptiveThinking: true and compat.supportsStrictTools: true; the supportsMidConvo* flags make no difference on pi-ai 0.84.x).
  2. npx --yes @keygraph/shannon@3.5.0 start -u <url> -r <repo> --models-config models.json --yes with SHANNON_AI_MODEL=anthropic:claude-mythos-5-1 against a mid-size Go + TypeScript monolith (pre-recon takes about 40 minutes).
  3. Wait for the pre-recon agent to finish its sub-agents and call the summary tools.

Expected behaviour

Pre-recon completes and the scan continues (or the model refuses the live phase, as it did on 2026-10-15), with the usual deliverables.

Actual behaviour

Run 1, Shannon 3.4.0:

[pre-recon] [ERROR] AGENT_EXECUTION_FAILED (unknown, attempt 1, 39m 35s, 13 turns)
[AGENT] pre-recon: Failed (AGENT_EXECUTION_FAILED) (39m 35s, $108.6530)
Scan FAILED   Status: failed   Duration: 41m 2s   Error: The agent could not complete its work. AGENT_EXECUTION_FAILED

Run 2, Shannon 3.5.0 (image keygraph/shannon:3.5.0), same target, same config:

[pre-recon] [ERROR] AGENT_EXECUTION_FAILED (unknown, attempt 1, 33m 38s, 16 turns)
[AGENT] pre-recon: Failed (AGENT_EXECUTION_FAILED) (33m 38s, $122.2308)
Scan FAILED

error.log in both runs: {"agent":"pi-executor","error":{"code":"AGENT_EXECUTION_FAILED","category":"unknown"},"turns":13,"retryable":false,"provider":{"provider":"anthropic","model":"claude-mythos-5-1","stopReason":"error", ...}} with the 400 above as errorMessageSnippet (run 2: req_011CgjzGButNRtwrnP1APEhU, 16 turns).

Pre-submission checklist (required)

  • I have searched the existing open issues and confirmed this bug has not already been reported.
  • I am running the latest released version of shannon. (Reproduced on 3.5.0; 3.4.0 fails the same way, both pin @earendil-works/pi-ai ^0.84.4.)

If applicable

  • I have included relevant error messages, stack traces, or failure details.
  • I have checked the workspaces folder for logs and pasted the relevant errors.
  • I have inspected the failed Temporal workflow run and included the failure reason.
  • I have included clear steps to reproduce the issue.
  • I have redacted any sensitive information (tokens, URLs, repo names).

CLI mode

npx

Provider

anthropic

Full command with all flags used (with redactions)

npx --yes @keygraph/shannon@3.5.0 start -u <url> -r <repo> --config <ci-config.yaml> --models-config models.json --workspace <name> --yes with SHANNON_AI_MODEL=anthropic:claude-mythos-5-1 and SHANNON_AI_API_KEY set in the environment (GitHub Actions, self-hosted runner).

OS (with version)

Ubuntu, GitHub Actions self-hosted runner (Actions Runner Controller)

Node.js version ('node -v')

v22.23.4

Docker version ('docker -v')

Docker Engine on the runner (version not captured in the run log), Docker Compose v5.5.1, image keygraph/shannon:3.5.0 (run 1 on 3.4.0)

Additional context

Opus 4.8 is not affected (no preserved thinking), full scans on it still complete. Possible fixes on the Shannon side: move the worker to pi-ai 1.5.x, or send thinking.block_binding.prefix_mismatch_behavior: "drop_block" with the thinking-binding-controls-2026-08-01 beta header for preserved-thinking models until then. Happy to test a build.

Activity

0

No activity yet.