Writable HTTP startup exits 2 while a peer holds the writer lease, so the Restart=always from #2203 becomes an unbounded restart loop

What happened?

A writable HTTP server refuses to start whenever any other process holds the palace writer lease, and it signals that refusal with SystemExit(2):

# mempalace/mcp_server/runtime.py:278-282  (_run_http_loop, develop @ 38260df)
if not _READ_ONLY:
    writer_ok, writer_reason = _acquire_mcp_writer_lock()
    if not writer_ok:
        logger.error("Writable MCP HTTP startup refused: %s", writer_reason)
        raise SystemExit(2)

Since #2203, deploy/mempalace-server.service:30-31 sets Restart=always with RestartSec=2. The two combine badly: a contended start is not a transient condition the process can recover from, because it exits before binding and re-acquires nothing on the way back up. The unit therefore restarts every few seconds indefinitely — it never starts, never gives up, and never reaches failed, so systemctl is-active reports activating forever and nothing in the unit's state says why.

systemd's own start limiter does not catch this, because the cycle is too slow to trip it. Each attempt lives roughly a second (interpreter start, imports, lease probe, exit), so with the template's RestartSec=2 that is about 3.3 starts per 10 s against the default StartLimitBurst=5 / StartLimitIntervalSec=10s. On my unit (RestartSec=5, ~6.5 s per cycle) the restart counter climbed monotonically past 27 before I intervened:

sep 14 05:21:51 systemd[1166]: mempalace-hub.service: Scheduled restart job, restart counter is at 18.
sep 14 05:21:52 mempalace[780941]: Writable MCP HTTP startup refused: another mempalace writer already
    holds the palace lock for '/home/emil/.mempalace/palace': palace ... is held by PID 88779
    (/home/emil/.local/bin/mempalace-mcp); wait for it to finish or stop the holder before retrying
sep 14 05:21:57 systemd[1166]: mempalace-hub.service: Scheduled restart job, restart counter is at 19.
sep 14 05:22:03 systemd[1166]: mempalace-hub.service: Scheduled restart job, restart counter is at 20.

Each cycle pays a full interpreter and import startup (~1.6 s CPU here) to fail the same way.

Why this state is reachable in normal operation

This is not an exotic race. It is the successor state to #2201, and the fix for #2201 is what makes it visible.

When the HTTP server is down — exactly the situation #2201 describes, and still the default outcome for any deployment whose env file predates #2203 — clients fall back to direct palace access, and the first session to call a mutating tool takes the process-lifetime writer lease held until atexit. So by the time an operator notices the server is gone and restarts it, a peer very often holds the lease. Before #2203 the unit stayed down silently; after #2203 it restart-loops silently. The failure is quieter than the one it replaced, because activating (auto-restart) looks more like progress than inactive (dead) does.

Field data from a single-user local hub on 3.9.0, where the whole chain ran end to end:

• 09-11 05:30:50 — the server exited via the idle watchdog (MCP server idle for 8.0 h (limit 8.0 h); exiting to release file handles.). My unit was hand-written and predated #2203, so Restart=on-failure ignored the clean exit and it simply stayed down. That part is #2201 exactly.
• It remained down for 2 days 8 hours without anyone noticing.
• Meanwhile every session reverted to serving the palace in-process (~900 MB–1 GB RSS each). One session's MCP server took the writer lease at 09-13 21:26 and held it. Other sessions' writes were refused with Peer MCP writer active; this server is read-only for mutating tools, and unrelated background work in those sessions was OOM-killed by the host harness.
• On restart, the loop above: 27 refused starts, resolved only by SIGTERMing the lease holder by hand and then starting the unit.

The operator-visible lesson is that stop unit → kill the holder → start unit is the only order that works, and nothing in the logs or the unit state suggests it.

Related, same exit path, not covered by #2203

The idle watchdog terminates with os._exit(0) (mempalace/mcp_server/protocol.py:1207), which bypasses both registered atexit handlers: server_registry.clear_serverinfo (mempalace/mcp_server/http.py:1162) and _release_mcp_writer_lock (mempalace/mcp_server/_guards.py:560). The advisory lock is reclaimed by the kernel when the process dies, so the lease itself is fine, but serverinfo.json keeps the dead PID. In my case it advertised pid 1950581 for the full 2 days 8 hours after that process exited. Clients tolerate this because they check PID liveness before dialing, so the impact is small — but sys.exit(0) in place of os._exit(0) would clear the record and cost nothing, and the watchdog is not in a context where atexit is unsafe to run.

What did you expect?

One of two things, rather than an invisible loop:

1. The server waits for the lease instead of exiting. Its own error message already prescribes this — "wait for it to finish or stop the holder before retrying" — and the holder is transient by design, since a session's MCP server releases at process exit. A singleton writable hub is the intended owner of the palace; treating contention as fatal at startup rather than as something to wait through is what forces manual intervention.
2. Failing that, it fails in a way a supervisor can act on: the unit lands in failed and stops re-executing, so systemctl status shows a terminal state with the refusal reason attached.

How to reproduce:

Minimal, no idle wait needed — two writable HTTP servers on the same palace:

1. mempalace serve --host 127.0.0.1 --port 8765 and let it acquire the lease.
2. Install deploy/mempalace-server.service (Restart=always, RestartSec=2) pointed at the same palace on a different port, then systemctl start mempalace-server.
3. journalctl -fu mempalace-server shows Writable MCP HTTP startup refused: ... and Scheduled restart job, restart counter is at N climbing without bound. systemctl is-active stays activating; the unit never reaches failed.

To reproduce the full chain including the entry condition, set MEMPALACE_MCP_IDLE_HOURS=0.02 per #2201, let the server self-exit after ~72 s, make one mutating MCP call from any client so that client takes the lease, then start the unit.

Environment:

• OS: Ubuntu (systemd 255, user unit with lingering enabled)
• Python version: 3.12
• MemPal version: released 3.9.0 (pipx); line references above traced on develop @ 38260df. The refusal (runtime.py:279-282), the watchdog arming in _run_http_loop (runtime.py:291), the 8.0 h default (_guards.py:114) and the os._exit(0) (protocol.py:1207) are all present on both.

Suggested fix

I would rather ask than assume here, since the first option is a core behaviour change and CONTRIBUTING asks for that to be raised before it is sent.

The option I would pick is a bounded wait in _run_http_loop: retry _acquire_mcp_writer_lock() with backoff for a configurable window before giving up, logging once at the start of the wait so the reason is visible. That matches what the error text already tells the operator to do, and it makes the common case — a session server that will exit on its own — resolve without intervention.

If touching the core is unwanted, the deployment-only fix is to widen the rate-limit window in deploy/mempalace-server.service until the burst is actually reachable — StartLimitIntervalSec=300 with StartLimitBurst=5, say — so the loop terminates in failed rather than running forever. (StartLimitIntervalSec=0 would be the wrong direction: it disables rate limiting entirely.) That is strictly worse for the user, since a lease held by a peer that later exits would then need a manual start, but it at least makes the condition legible.

Independently of which is chosen, sys.exit(0) instead of os._exit(0) in the idle watchdog would let the serverinfo record clear itself.

I am happy to implement either. This also bears on the broader question left open at the end of #2201 — whether _run_http_loop should arm the idle watchdog at all, given that #1552's stated purpose is reaping per-session stdio servers. The 2-day outage above is a data point that it should not: on a singleton hub the watchdog has no stale-handle problem to solve, and its only observed effect was to take the palace's single writer offline overnight and hand the lease to a random session.