Hermes PMA turn stalls indefinitely — ACP completion notification dropped after subagent delegation

Bug

A hermes PMA managed thread stalls indefinitely in running state after the hermes agent successfully completes its response. The ACP prompt/completed or turn/completed notification is never delivered to the CAR hub, so the turn is never finalized. The thread appears likely_hung with no progress events, but the hermes session file shows the assistant fully completed its work.

Reproduction / Evidence

Thread: ae498edf-d39f-4599-8788-246a6913bb90 (hermes PMA, discord-bound) Backend session: e22870b5-fd95-46a6-bd10-cf88193f94ec Stalled turn: e8686d6f-3e11-47e0-b043-2488a80dd23f (backend turn-11) Version: codex-autorunner 1.9.17 Model: glm-5-turbo via ACP (hermes-m4-pma profile) Stalled since: 2026-04-11T14:37:05Z (1.5+ hours with no resolution)

Timeline

Hub log (smoking gun)

Stalled turn (turn-11):

raw_events_received=1, run_events_dispatched=0
last_runtime_method="prompt/started"

Previous successful turn (turn-8) for comparison:

raw_events_received=57, run_events_dispatched=21
last_runtime_method="prompt/completed"

Hermes session file confirms completion

The hermes session (session_e22870b5-*.json) contains 136 messages. The last assistant message (msg 135) is a complete response ending with "Want me to file this as a codex-autorunner issue?" — the assistant finished its work, but the ACP completion notification never reached CAR.

DB state

-- Turn never finalized:
status = 'running', finished_at = NULL, model = NULL, assistant_text = NULL, error = NULL
-- Thread still shows:
normalized_status = 'running', status_reason_code = 'turn_started'

Root Cause Analysis

The hermes agent completed its response (session file updated, 136 messages including the final assistant reply), but no ACP lifecycle event (prompt/completed, turn/completed, or session/update with terminal status) was emitted after the initial prompt/started.

Most likely trigger: The turn involved a delegate_task (subagent) call. The hermes agent spawned a subagent that completed successfully (visible in msg 134 as a tool result), then produced its final assistant text (msg 135). It is plausible that the subagent delegation path has a code path where the ACP completion notification is dropped — either:

1. The hermes ACP adapter's prompt/completed emission is skipped when the turn involves subagent delegation (e.g., the turn completion handler does not fire because control returned through a non-standard path).
2. An unhandled exception in the post-response hook (e.g., token counting, session persistence) silently swallows the completion notification before it is written to the ACP stdout/stderr transport.

Why this is hard to detect:

• The hermes session file shows a complete, successful conversation — no error.
• CAR's progress pump receives only prompt/started and waits for a terminal event that never comes.
• No error is logged anywhere — the turn just silently stops emitting events.
• The thread status remains running forever, eventually showing as likely_hung.

Impact

• PMA threads using hermes via ACP can silently stall after any turn that involves subagent delegation.
• No automatic recovery — the thread stays running until manually interrupted.
• The PMA's response is lost (never delivered to the Discord channel) because turn_finalized never fires.
• This is distinct from the model simply being slow — the model completed but the completion signal was lost.

Suggested Fix

1. Add a watchdog in the hermes ACP adapter that detects when the hermes process has finished writing to the session file but no prompt/completed has been emitted — synthesize a completion event.
2. Add logging in the ACP transport layer for all lifecycle events to make it possible to trace where the notification is dropped.
3. Consider adding a session-file mtime-based stall detection: if the session file has not been updated for N seconds after a prompt/started, mark the turn as stalled rather than leaving it in running indefinitely.

Related

• #1365 — car pma thread send false timeout on successful delivery (different ACP transport issue)