Medic doesn't detect persistently 'pending' steps (cron fires but sub-agent spawn fails silently)

Open

robobobbyopened 8mo ago

Problem

The medic currently catches steps stuck in 'running' state (ghost claims from crashed agents) and zombie runs. But it doesn't detect steps stuck in 'pending' state where crons fire correctly but sub-agent spawning silently fails.

Observed pattern:

Step transitions to 'pending'
Agent cron fires (every 5 min), calls step peek → 'HAS_WORK'
Agent cron calls step claim → gets step JSON
Agent cron should spawn a sub-agent session
Sub-agent spawn fails silently (no error logged, no session created)
Step stays 'pending' indefinitely (cron fires again, same result)

The medic only watches for status = 'running' steps. A step that's stuck in status = 'pending' for 30+ minutes is invisible to the current health checks.

Suggested Fix

Add a new medic check: stale_pending_steps

Find steps with status = 'pending' where updated_at is older than 2x the cron interval (>10 min)
Log as a warning finding (don't reset — the step is safe as-is)
Alert via main session so the operator can investigate or manually claim/complete

The check shouldn't auto-reset pending steps because there's no risk of data corruption — only ghost 'running' steps need resetting.

Environment

Antfarm v0.5.1
Observed during feature-dev, bug-fix, and security-audit workflow test run

Activity

Keesan123mo ago

Interesting failure mode. The main thing I would separate here is:

step is pending because nothing has tried to start yet
step is pending because spawn is in-flight
step is pending because spawn never actually produced a child session

Those look identical if you only watch step state, but operationally they are very different.

One pattern that has helped in practice is to require a positive child-run proof before a pending step is allowed to stay healthy for long:

pending + no spawn attempt yet = normal
pending + spawn attempt recorded but no child session heartbeat / receipt within TTL = explicit failure state
pending + child session exists = healthy in-flight

That gives you a cleaner split between queueing and execution. Otherwise the system can look alive because cron keeps firing even though no real worker ever came into existence.

I’d probably add two concrete things beyond the warning:

a monotonic spawn_attempt_count + last_spawn_attempt_at
a typed reason like spawn_missing, spawn_failed, or child_session_not_observed

That makes the medic output much more actionable than just “stale pending”.

We ended up caring about exactly this class of false-progress problem in MartinLoop, because some loops looked active from the outside while no trustworthy child-run evidence existed underneath. If this framing is useful, I’d genuinely love your take on whether the boundary should be pending -> failed or pending -> needs_attention first.