#205

Keep the server up when it restarts with background work running

Opencalledforthwants to merge intofrom

What happened

When the server restarts while a session still lists background tasks (typical for an agent left running overnight), startup clears those tasks and announces the change. The announcement went out while the server was still waiting to listen. publish() checked the event's environment by calling bootstrap(), which validates the whole bootstrap response, and websocketUrl is only set after listen. The ZodError was thrown inside a microtask, so it killed the process. Under node --watch the server then sat at "Waiting for file changes before restarting".

A restart is easy to trigger in development: pnpm install in any worktree relinks files in the main checkout's node_modules, and node --watch restarts on that.

Changes

  • Nothing is sent before the server listens. publishDurableEvent and publishThreadEvent are bound only after listen. Nobody can be connected earlier, and durable events are stored and replay on reconnect.
  • The environment check no longer goes through bootstrap(). attachWebSocket takes environmentId directly for subscribe, replay and publish. This also stops every published event from rebuilding and validating the provider list.
  • A failed live send is logged, not thrown. The event is already stored, so the cost is a live delivery, not the process.
  • Crashes reach the service log. When a log file is configured, uncaughtExceptionMonitor writes the crash and its stack there. The supervisor only records an exit code, and a headless service has no console. The monitor doesn't change how the process exits.

Tests

  • New server.test.ts case: start, stop, leave a session running with a background task in SQLite, start again. Before the fix it fails with the same websocketUrl ZodError seen in the field. After the fix it starts, serves /health, and the session reads idle with no tasks.
  • pnpm --filter @openmanager/server test: 780 passed, 7 skipped. Typecheck and lint are clean.

Known, not changed here

From an audit of other ways the process can die:

  • threadService.onRuntimeEvent has no guard. For Claude, a thrown projection error is treated as the CLI dying, so the session ends. The server stays up.
  • ACP child processes only listen for 'error' while spawning.
  • The WebSocket upgrade handler touches SQLite (audit, authenticate) with no guard.
  • A throwing after-reply task on an async command becomes an unhandled rejection.

🤖 Generated with Claude Code

Activity

9
greptile-apps[bot]23d ago


[Medium risk] Adjusts server startup and event handling for graceful restarts.

The PR appears safe to merge; no blocking issues remain.

Summary

Keeps the server running when startup clears background work left by the previous process.

  • Binds live event publication after the server listens.
  • Checks event environments without rebuilding bootstrap().
  • Logs failed durable-event sends instead of letting them crash the process.
  • Records crash details in file logs and removes recognized secrets from crash and startup messages.
  • Adds restart and child-process crash tests. The crash test addresses the previous unnumbered finding.

No new actionable issues were found in the changes since the last review.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[Restore saved sessions] --> B[Clear abandoned background work]
  B --> C[Store durable events]
  C --> D[Wait for listener]
  D --> E[Bind live publication]
  E --> F[Clients connect and read saved state]
  E --> G[Publish later events]
  G --> H{Send succeeds?}
  H -->|Yes| I[Client receives event]
  H -->|No| J[Log failure; server stays running]

Reviews (3) · Last reviewed commit: "fix(server): redact crash lines, and say..." · Reviewed by Greptile

greptile-apps[bot]reviewed23d ago
greptile-apps[bot]reviewed23d ago
calledforthreviewed23d ago
calledforthreviewed23d ago