#467
Swarm hosts get no runaway-memory protection: one uncapped agent child + a slice MemoryHigh = whole-host reclaim livelock (nothing ever kills it)
(Fable 5.1:)
Summary
acfs sizes and launches multi-agent swarms (
acfs capacity,acfs swarm plan, NTM panes) but provisions the host with no mechanism that ends a memory runaway:
- no userspace OOM killer: Ubuntu server images do not seed
systemd-oomd(apt-cache show systemd-oomd→Priority: optional,Task:lists only desktop flavours;systemctl is-enabled systemd-oomd→not-foundon a stock 26.04 VPS), andinstall.shneither installs nor configures one (install.sh:8239package list; zero hits foroomd|ManagedOOM|earlyoom|oom.groupanywhere in the repo history atee31a47b);- no per-agent-scope
MemoryMax: the opt-in resource profile (scripts/lib/capacity.sh:335-339) sets onlyCPUWeight/IOWeight/TasksMax+ accounting; tmux 3.6 (libsystemd) puts every pane in its owntmux-spawn-<uuid>.scopewithmemory.max=max;- no swap-headroom or pressure preflight:
capacity.shreads onlyMemTotal(:173),swarm_doctor.sh'sresource_pressurecheck (:288-297) usesMemAvailable, load and disk — a host with swap 100 % full passes;- and the isolation design doc recommends
MemoryHigh=three times (docs/operations/swarm-resource-isolation.md:12, 40, 128) without ever pairing it with something that kills on breach.Result on a real swarm host (92 GiB / 24 cores / 32 GiB swap, Ubuntu 26.04, systemd 259, kernel 7.0, 30–45 panes): three whole-host livelocks in six weeks. Each time swap was already 100 % used, one agent pane launched an uncapped multi-GB child (a repo doctor script at 15 GB then 34 GB; a storm of
git maintenancerepacks; an indexer run) inside its plain tmux scope, the user slice crossed aMemoryHigh=that users had added for "protection", and every allocation on the host spun in reclaim: load 400+, 95 % sys, PSI memory full 80–99 %, every agent pane and two bitcoind nodes in D-state (folio_wait_bit_common), 3 GB/s of page-cache re-reads. Nothing fired. One episode lasted six hours until a human noticed; relief was always a manualkillof the child (MemAvailable 5 → 36 GB within seconds).A cap without a killer is the failure mode, so the ask is explicitly not "add
MemoryHigh". The ask is that acfs, which owns the swarm host's capacity model and the opt-in isolation profile, (1) gives every agent scope a hardMemoryMaxceiling, (2) installs/configuressystemd-oomdon the agent slice as the slice-level safety net, (3) checks swap headroom and PSI before recommending a swarm size, and (4) fixes the doc so noMemoryHigh=is recommended without one of the two killers.Why
MemoryHighalone livelocks (docs)
- Kernel
admin-guide/cgroup-v2,memory.high: "If a cgroup's usage goes over the high boundary, the processes of the cgroup are throttled … Going over the high limit never invokes the OOM killer and under extreme conditions the limit may be breached."memory.max: "If a cgroup's memory usage reaches this limit and can't be reduced, the OOM killer is invoked in the cgroup."systemd.resource-control(5),MemoryHigh=: "Memory usage may go above the limit if unavoidable, but the processes are heavily slowed down and memory is taken away aggressively."MemoryMax=: "If memory usage cannot be contained under the limit, out-of-memory killer is invoked inside the unit. It is recommended to use MemoryHigh= as the main control mechanism and use MemoryMax= as the last line of defense."- Throttling is hierarchical: once a slice is over
highwith nothing reclaimable (swap full orMemorySwapMax=0, anon footprint above the cap), every task charging memory under that slice is slowed and reclaim keeps scanning page cache forever. That is the 95 % sys / D-state signature.systemd-oomd.service(8): "It is highly recommended for the system to have swap enabled for systemd-oomd to function optimally … Without swap, the system enters a livelocked state much more quickly and may prevent systemd-oomd from responding in a reasonable amount of time." A host whose swap is 100 % used is, for this purpose, a host without swap — which is why a swap-headroom preflight belongs in the capacity model.Reproduction (plain Ubuntu 24.04/26.04 VM, root, < 1 GiB RAM, ~4 min)
Three phases in one self-contained script. Phase A is the bug; B and C are the two fix layers.
MemorySwapMax=0on the slice models "swap already full".Save as
repro.sh, thensudo bash repro.sh all(ornokill/oomd/scopemaxindividually,cleanupat the end):#!/usr/bin/env bash # memory.high without a killer => reclaim livelock; memory.high + systemd-oomd => contained kill # # Self-contained. Run as root on a plain Ubuntu 24.04/26.04 VM (cgroup v2 + PSI, # systemd >= 250). Nothing here is distribution-specific beyond `apt-get install # systemd-oomd` in the oomd phase. Uses < 1 GiB of RAM; no swap is touched # (MemorySwapMax=0 on the slice models a host whose swap is already 100% full). # # sudo bash repro.sh nokill # phase A: slice MemoryHigh, no killer -> crawl, no kill # sudo bash repro.sh oomd # phase B: same + systemd-oomd ManagedOOMMemoryPressure=kill -> a descendant scope is killed (note the order) # sudo bash repro.sh scopemax # phase C: same + MemoryMax on the runaway's own scope -> instant contained kernel kill, neighbour untouched # sudo bash repro.sh both # A then B (default); all = A, B, C # sudo bash repro.sh cleanup # stop units, revert runtime properties, remove files # # Tunables (env): HIGH=256M RUN_FOR=90 RUNAWAY_MIB=768 CACHE_MIB=128 set -euo pipefail PHASE="${1:-both}" SLICE=repro-agents.slice HIGH="${HIGH:-256M}" RUN_FOR="${RUN_FOR:-90}" RUNAWAY_MIB="${RUNAWAY_MIB:-768}" CACHE_MIB="${CACHE_MIB:-128}" DIR=/var/tmp/repro-livelock # must be disk-backed (tmpfs pages are not reclaimable with swap off) CG="" # slice cgroup dir, resolved from the running victim unit say() { printf '\n[%s] %s\n' "$(date -u +%H:%M:%S)" "$*"; } write_helpers() { mkdir -p "$DIR" # Runaway "agent child": grows anonymous memory in 16 MiB steps and never # exits (a repo doctor script, an indexer, a repack...). Prints the wall time # of each step so the throttling crawl is visible in the journal. cat > "$DIR/alloc.py" <<'PY' import sys, time total_mib = int(sys.argv[1]); step_mib = 16 chunks = []; held = 0; t0 = time.monotonic() while held < total_mib: ts = time.monotonic() buf = bytearray(step_mib * 1024 * 1024) for off in range(0, len(buf), 4096): buf[off] = 1 chunks.append(buf); held += step_mib print(f"alloc +{step_mib} MiB -> {held} MiB held, step took {time.monotonic()-ts:.3f}s (elapsed {time.monotonic()-t0:.1f}s)", flush=True) print(f"holding {held} MiB", flush=True) while True: ts = time.monotonic() for buf in chunks: for off in range(0, len(buf), 1 << 20): buf[off] ^= 1 print(f"touch pass took {time.monotonic()-ts:.3f}s", flush=True) time.sleep(2) PY # Well-behaved neighbour: every other agent pane in the slice. Owns a 32 MiB # file (its "working set") and re-reads it once a second, plus a small # allocation. Under a slice-wide memory.high breach its file pages are # reclaimed as fast as they are faulted in, so each pass turns into disk I/O. cat > "$DIR/victim.py" <<'PY' import os, sys, time path = sys.argv[1] if not os.path.exists(path): with open(path, "wb") as f: f.write(os.urandom(32 * 1024 * 1024)) t0 = time.monotonic() while True: ts = time.monotonic() with open(path, "rb") as f: while f.read(1 << 20): pass t_read = time.monotonic() - ts ts = time.monotonic() b = bytearray(8 * 1024 * 1024) for off in range(0, len(b), 4096): b[off] = 1 del b print(f"victim: read 32 MiB working set {t_read:.3f}s, 8 MiB alloc {time.monotonic()-ts:.3f}s (elapsed {time.monotonic()-t0:.0f}s)", flush=True) time.sleep(1) PY } ev() { awk -v k="$2" '$1==k{print $2}' "$1/memory.events" 2>/dev/null || echo 0; } stat_() { awk -v k="$2" '$1==k{print $2}' "$1/memory.stat" 2>/dev/null || echo 0; } psi_full() { awk '/^full/{sub("avg10=","",$2); print $2}' "$1/memory.pressure" 2>/dev/null || echo 0; } ustate() { printf '%s/%s' "$(systemctl is-active "$1" 2>/dev/null || true)" "$(systemctl show -p Result --value "$1" 2>/dev/null)"; } resolve_cg() { local unit_cg unit_cg=$(systemctl show -p ControlGroup --value repro-victim.service) CG="/sys/fs/cgroup$(dirname "$unit_cg")" [[ -f "$CG/memory.high" ]] || { echo "cannot find slice cgroup at $CG" >&2; exit 1; } } stop_units() { systemctl stop repro-runaway.service repro-victim.service 2>/dev/null || true systemctl reset-failed repro-runaway.service repro-victim.service 2>/dev/null || true rm -f "$DIR"/cache.*.bin "$DIR"/victim.*.bin } start_workload() { # victim first so the slice exists, then apply the slice-level cap (what an # operator does when they put MemoryHigh= on a user/agent slice). systemd-run --quiet --unit=repro-victim --slice="$SLICE" --collect -p MemoryAccounting=yes \ python3 "$DIR/victim.py" "$DIR/victim.$$.bin" systemctl set-property --runtime "$SLICE" MemoryAccounting=yes MemoryHigh="$HIGH" MemorySwapMax=0 resolve_cg # The runaway scope: anon growth plus a page-cache churner. The churner # creates its file INSIDE the unit so the pages are charged to this cgroup # (page cache is charged to the first toucher). As in the real incident, # reclaim then has file pages to scan forever but can never get the slice # below memory.high, because the anon footprint alone exceeds it. systemd-run --quiet --unit=repro-runaway --slice="$SLICE" --collect -p MemoryAccounting=yes \ bash -c "head -c ${CACHE_MIB}M /dev/urandom > '$DIR/cache.$$.bin'; while :; do cat '$DIR/cache.$$.bin' >/dev/null; done & exec python3 '$DIR/alloc.py' $RUNAWAY_MIB" sleep 2 say "slice $SLICE at $CG: memory.high=$(cat "$CG/memory.high") memory.max=$(cat "$CG/memory.max") memory.swap.max=$(cat "$CG/memory.swap.max")" say "units: runaway=$(ustate repro-runaway.service) victim=$(ustate repro-victim.service)" } sample_loop() { local stop_when_runaway_dead="$1" t=0 h0 p0 h0=$(ev "$CG" high); p0=$(stat_ "$CG" pgscan) printf '\n%-4s %-9s %-9s %-8s %-8s %-10s %-16s %s\n' t_s slice_cur high_ev oom_kill psi_full pgscan runaway/victim victim_last_line while (( t <= RUN_FOR )); do local cur hi ok pf pg rs vs vl cur=$(( $(cat "$CG/memory.current") / 1048576 ))M hi=$(( $(ev "$CG" high) - h0 )); ok=$(ev "$CG" oom_kill); pf=$(psi_full "$CG"); pg=$(( $(stat_ "$CG" pgscan) - p0 )) rs=$(systemctl is-active repro-runaway.service || true); vs=$(systemctl is-active repro-victim.service || true) vl=$(journalctl -u repro-victim -o cat -n 1 --no-pager 2>/dev/null | tail -1) printf '%-4s %-9s %-9s %-8s %-8s %-10s %-16s %s\n' "$t" "$cur" "$hi" "$ok" "$pf" "$pg" "$rs/$vs" "$vl" if [[ "$stop_when_runaway_dead" == yes && "$rs" != active ]]; then break; fi sleep 5; t=$((t+5)) done say "last runaway log line: $(journalctl -u repro-runaway -o cat -n 1 --no-pager | tail -1)" } phase_nokill() { say "PHASE A - slice MemoryHigh=$HIGH, swap off, no OOM killer configured ($(systemctl --version | head -1), kernel $(uname -r))" write_helpers; stop_units systemctl set-property --runtime "$SLICE" ManagedOOMMemoryPressure=auto 2>/dev/null || true start_workload sample_loop no local rs; rs=$(systemctl is-active repro-runaway.service || true) say "RESULT A: after ${RUN_FOR}s runaway=$rs oom_kill=$(ev "$CG" oom_kill) high_events=$(ev "$CG" high) psi_full_avg10=$(psi_full "$CG")% pgscan=$(stat_ "$CG" pgscan)" if [[ "$rs" == active && "$(ev "$CG" oom_kill)" == 0 ]]; then echo " => the slice sits above memory.high indefinitely: the runaway is throttled to a crawl, reclaim scans continuously (pgscan), PSI full is pinned near 100%, and NOTHING kills anything. This is the livelock; at host scale the reclaim CPU/IO is what freezes every other pane." else echo " => unexpected: the runaway is not running (check journalctl -k for an OOM kill)." fi stop_units } phase_oomd() { say "PHASE B - same slice cap + systemd-oomd ManagedOOMMemoryPressure=kill" write_helpers; stop_units if [[ ! -x /usr/lib/systemd/systemd-oomd ]] && ! command -v systemd-oomd >/dev/null 2>&1; then say "installing systemd-oomd"; apt-get install -y -q systemd-oomd fi mkdir -p /etc/systemd/oomd.conf.d printf '[OOM]\nDefaultMemoryPressureLimit=50%%\nDefaultMemoryPressureDurationSec=10s\n' > /etc/systemd/oomd.conf.d/50-repro.conf systemctl enable systemd-oomd.service >/dev/null 2>&1 || true systemctl restart systemd-oomd.service systemctl set-property --runtime "$SLICE" ManagedOOMMemoryPressure=kill ManagedOOMMemoryPressureLimit=50% say "systemd-oomd: $(systemctl is-active systemd-oomd.service)" local since; since=$(date -u +%Y-%m-%dT%H:%M:%S) start_workload oomctl 2>/dev/null | grep -A4 "$SLICE" | head -6 || true sample_loop yes say "systemd-oomd kills since phase start (order matters):" journalctl -u systemd-oomd --since "$since" --no-pager -o short-iso | grep -E 'Killed .* due to' || echo " (none logged)" local rs; rs=$(systemctl is-active repro-runaway.service || true) say "RESULT B: runaway=$(ustate repro-runaway.service) victim=$(ustate repro-victim.service) slice_cur=$(( $(cat "$CG/memory.current") / 1048576 ))M psi_full_avg10=$(psi_full "$CG")%" if [[ "$rs" != active ]]; then echo " => systemd-oomd SIGKILLed whole descendant cgroups until slice pressure fell; the host never spun. NOTE the kill order above:" echo " oomd ranks candidates by reclaim-scan rate, then memory usage. With swap unavailable the anon hog is barely scanned," echo " so a neighbour with a hot file working set can be killed first. A per-scope MemoryMax= is the deterministic layer;" echo " oomd is the slice-level safety net that stops the livelock." else echo " => no kill within ${RUN_FOR}s - check 'oomctl' (is the slice listed with pressure?) and that /proc/pressure exists." fi stop_units } phase_scopemax() { # The deterministic layer: a hard MemoryMax on the runaway's OWN scope, sized # below the slice's MemoryHigh. The kernel memcg OOM killer fires inside that # scope only, instantly, and takes the largest task there (the hog). The slice # never crosses memory.high, so the neighbour never notices. This is what # `systemd-run --scope -p MemoryMax=` around an agent / agent child gives you. local SCOPE_MAX="${SCOPE_MAX:-192M}" say "PHASE C - same slice MemoryHigh=$HIGH, runaway scope additionally capped with MemoryMax=$SCOPE_MAX (no oomd needed)" write_helpers; stop_units systemctl set-property --runtime "$SLICE" ManagedOOMMemoryPressure=auto 2>/dev/null || true systemd-run --quiet --unit=repro-victim --slice="$SLICE" --collect -p MemoryAccounting=yes \ python3 "$DIR/victim.py" "$DIR/victim.$$.bin" systemctl set-property --runtime "$SLICE" MemoryAccounting=yes MemoryHigh="$HIGH" MemorySwapMax=0 resolve_cg local h0; h0=$(ev "$CG" high) systemd-run --quiet --unit=repro-runaway --slice="$SLICE" --collect -p MemoryAccounting=yes -p MemoryMax="$SCOPE_MAX" \ bash -c "head -c ${CACHE_MIB}M /dev/urandom > '$DIR/cache.$$.bin'; while :; do cat '$DIR/cache.$$.bin' >/dev/null; done & exec python3 '$DIR/alloc.py' $RUNAWAY_MIB" local t=0 rcg rcg="/sys/fs/cgroup$(systemctl show -p ControlGroup --value repro-runaway.service)" while (( t < 60 )) && [[ "$(systemctl is-active repro-runaway.service || true)" == active ]]; do sleep 1; t=$((t+1)); done say "after ${t}s: runaway=$(ustate repro-runaway.service) victim=$(ustate repro-victim.service)" say "slice: high_events_delta=$(( $(ev "$CG" high) - h0 )) psi_full_avg10=$(psi_full "$CG")% | kernel: $(journalctl -k --no-pager -o cat | grep -E 'Memory cgroup out of memory' | tail -1)" say "victim last line: $(journalctl -u repro-victim -o cat -n 1 --no-pager | tail -1)" say "RESULT C: the runaway was killed by the kernel inside its own scope (unit result above); the slice never breached memory.high, the neighbour was never throttled." echo " (A scope's default OOMPolicy=continue keeps the pane's shell alive and kills just the biggest task; OOMPolicy=kill / memory.oom.group=1 would take the whole scope.)" stop_units } cleanup() { stop_units systemctl set-property --runtime "$SLICE" MemoryHigh=infinity MemorySwapMax=infinity ManagedOOMMemoryPressure=auto ManagedOOMMemoryPressureLimit=0% 2>/dev/null || true systemctl stop "$SLICE" 2>/dev/null || true rm -f /etc/systemd/oomd.conf.d/50-repro.conf; rm -rf "$DIR" systemctl try-restart systemd-oomd.service 2>/dev/null || true say "cleaned up (systemd-oomd left installed; disable with: systemctl disable --now systemd-oomd)" } case "$PHASE" in nokill) phase_nokill ;; oomd) phase_oomd ;; scopemax) phase_scopemax ;; both) phase_nokill; phase_oomd ;; all) phase_nokill; phase_oomd; phase_scopemax ;; cleanup) cleanup ;; *) echo "usage: $0 nokill|oomd|scopemax|both|all|cleanup" >&2; exit 2 ;; esacObserved output (Ubuntu 24.04, systemd 255.4, kernel 7.0; identical shape on systemd 259)
Phase A — slice
MemoryHigh=256M, swap unavailable, no killer. The runaway sits abovehighfor the whole run; its 16 MiB steps take 28–48 s; the slice'smemory.events highclimbs by ~19,000,pgscanby 5.8 M, PSI memory full avg10 ≈ 99 %;oom_killstays 0; both units are stillactivewhen the script gives up:t_s slice_cur high_ev oom_kill psi_full pgscan runaway/victim 0 263M 0 0 7.42 0 active/active 30 282M 3409 0 94.03 2121127 active/active 60 287M 4804 0 98.45 2849505 active/active 90 289M 5937 0 98.97 3313402 active/active last runaway log line: alloc +16 MiB -> 272 MiB held, step took 27.952s (elapsed 31.0s) RESULT A: after 90s runaway=active oom_kill=0 high_events=19012 psi_full_avg10=98.98% pgscan=5837074Scale this from 256 MiB to a 78 GiB slice with 40 panes and it is the incident: nothing ends it.
Phase B — same, plus
systemd-oomdwithManagedOOMMemoryPressure=killon the slice (DefaultMemoryPressureLimit=50%,DefaultMemoryPressureDurationSec=10s). oomd SIGKILLs descendant cgroups until the slice's pressure drops, and the livelock ends within the configured window:systemd-oomd: Killed /repro.slice/repro-agents.slice/repro-victim.service due to memory pressure for /repro.slice/repro-agents.slice being 82.87% > 50.00% for > 10s with reclaim activity systemd-oomd: Killed /repro.slice/repro-agents.slice/repro-runaway.service due to memory pressure for /repro.slice/repro-agents.slice being 94.32% > 50.00% for > 10s with reclaim activity RESULT B: runaway=inactive/success victim=inactive/success slice_cur=0MNote the order: in 3 of 3 runs oomd killed the well-behaved neighbour first and the hog ~15 s later.
oomd_select_by_pgscan_rate()(src/oom/oomd-util.c) ranks candidates by reclaim-scan delta, then memory usage; with swap unavailable the hog's anonymous pages are never scanned while the neighbour's hot file working set is scanned on every pass. On a swarm host the "neighbours" are the panes and daemons with large page-cache working sets. oomd is the right slice-level net (it stops the host from spinning) but not a precise first line.Phase C — same slice cap, plus
MemoryMax=192Mon the runaway's own unit (whatsystemd-run --scope -p MemoryMax=around an agent or agent child provides). The kernel memcg OOM killer acts inside that scope within one second, takes the largest task there (the hog), the slice never crosseshigh, and the neighbour never notices:after 1s: runaway=inactive/success victim=active/success slice: high_events_delta=0 psi_full_avg10=1.59% kernel: Memory cgroup out of memory: Killed process … (python3) … anon-rss:193776kB victim last line: victim: read 32 MiB working set 0.008s, 8 MiB alloc 0.011sThe same containment was observed in production on the affected host: the one pane that did run under a
systemd-run --scope -p MemoryMax=32768Mwrapper had two 5.8 GB python children memcg-killed (oom-kill:constraint=CONSTRAINT_MEMCG … oom_memcg=…/run-p….scope) without anyone else noticing, while the 43 uncapped panes took the host down.Where acfs stands today (
origin/main@ee31a47b, v0.10.0; the cited files are unchanged since83daee01)
layer status userspace OOM killer ( systemd-oomd,ManagedOOM*)absent from installer, docs, doctor; never mentioned in repo history per-scope hard ceiling ( MemoryMax) on agent scopesabsent; doc :13/:68advises against it for model CLIs (fine) but offers no alternative for their childrenswap headroom / PSI in capacity or doctor absent ( capacity.sh:173readsMemTotalonly;swarm_doctor.sh:288-297usesMemAvailable, load, disk)MemoryHighguidancerecommended at swarm-resource-isolation.md:12, 40, 128with no pairing killerNTM (installed by acfs) wraps only the Claude template in systemd-run --user --scope -q -p MemoryMax=(internal/config/templates.go:609, 25 % of RAM clamped 2–16 GiB); Codex, Gemini, Antigravity, Grok, Cursor, Ollama templates have no memory property; custom launchers and relaunch paths lose the prefixSuggested fix shape (layered; each layer is independently useful)
- Hard per-scope ceiling for every agent class. In the opt-in profile's
acfs-scope/ccs/cods/gmiswrappers (capacity.sh:415-457) add-p MemoryMax=<N>sized from the capacity model (the NTM formula, 25 % of effective RAM clamped, is a reasonable default; make it a profile value), for all classes — this is a ceiling on the pane's process tree, so it protects against children regardless of which CLI spawned them. Leave scopes at their defaultOOMPolicy=continue: the kernel then kills the largest task in the scope (the runaway child), the agent CLI survives and sees the failed command. UseOOMPolicy=kill(=memory.oom.group=1,systemd.scope(5)) only foracfs-background.slice/acfs-local-build.slicework where "all or nothing" is correct. Recommend the same prefix for the non-Claude NTM templates. This is the deterministic layer (phase C).systemd-oomdas the slice-level net. Installsystemd-oomd(it is one apt package on Ubuntu, not seeded on server), enable it, ensureDefaultMemoryAccounting=yes(default), and ship a drop-in for the agent slice —acfs-agent.slicewhen the profile is applied, otherwiseapp.slice/app-tmux.sliceunder the user manager — withManagedOOMMemoryPressure=killandManagedOOMMemoryPressureLimit=40%(systemd-oomd.service(8)Usage Recommendations: "something likeuser@$UID.servicemay prefer a much lower value like 40%"; default is 60 %,oomd.conf(5)).ManagedOOMMemoryPressure=killmeans "systemd-oomd will select a descendant cgroup and send SIGKILL to all of the processes under it" (systemd.resource-control(5)); only leaf cgroups (each tmux pane scope) are candidates. Document the selection caveat from phase B and that it needs PSI + cgroup v2 + accounting. Surfaceoomctlinacfs doctor.- Preflight in
acfs capacity/swarm_doctorresource_pressure: readSwapTotal/SwapFreeand/proc/pressure/memory;warnwhen swap is > 80–90 % used or memory PSIfull avg60> 10 %,fail(zero recommended agents) when swap is 100 % used on a host with a cap or with oomd (zero headroom was the precondition of all three incidents); add an informational finding "no userspace OOM killer and no per-scope memory ceiling — overload will stall, not kill". OptionallyMemorySwapMax=as a proportional share per agent scope so one runaway cannot consume the host's entire swap (not0, which just pins it in RAM).- Docs. In
swarm-resource-isolation.md, state explicitly next to lines 12/40/128 thatMemoryHigh=on any slice must be paired withsystemd-oomdon that slice or replaced byMemoryMax=on its leaves, because breach ofhighnever invokes the OOM killer; and add "a swap-full host behaves like a swapless host for oomd" to the capacity notes.Workarounds today
sudo apt install systemd-oomd && sudo systemctl enable --now systemd-oomd, thensudo systemctl set-property user-<uid>.slice ManagedOOMMemoryPressure=kill ManagedOOMMemoryPressureLimit=40%(or a[Slice]drop-in),oomctlto confirm. Expect the first victim under swap-full conditions to be a page-cache-heavy neighbour.- Launch agents (all kinds, not just Claude) via
systemd-run --user --scope -q -p MemoryMax=<25 % of RAM> <cli> …, or run known-heavy helpers under such a scope.- Keep swap below ~80 % used; never add a bare
MemoryHigh=to a user/agent slice.Severity / scope
Whole-host hang affecting every agent on the machine; recurring (3× in six weeks on one host); recovery requires a human to find and kill the right process while
sshtakes 20 s to connect andsystemctl --usertimes out. Any swarm host with a sliceMemoryHigh=(or that simply exhausts swap) and an uncapped multi-GB child is exposed; the larger the swarm, the more likely one pane launches such a child. Reproduction artifacts available.
Activity
0No activity yet.