#327
Issues/omacom/try-omarchy/#327

Guest hard-freezes / RCU stalls from virtio-blk I/O hangs and vCPU starvation — escalating frequency

OpenyanUEdopened 14d ago

Related to (but distinct from) #326 (audio-bridge crash-loop, now disabled on my VM). This is a separate, older problem: the guest repeatedly hard-freezes with RCU stalls whose stacks point at the virtio-blk path or pure CPU starvation, and it is getting worse.

Environment

  • Try Omarchy on macOS (Apple Silicon), guest = Arch Linux ARM, kernel 7.2.6-1-aarch64-ARCH
  • 8 vCPU, 7.7 GiB RAM + 7.7 GiB zram, single virtio_blk root disk (vda, 24 GiB)
  • 9p shared folder mounted at /mnt/mac

Symptoms

Recurring stalls: guest stops making progress for minutes → rcu_preempt stall reports → systemd services are killed by their watchdogs (systemd-journald, systemd-logind, systemd-resolved, systemd-timesyncd, systemd-userdbd, systemd-udevd) → desktop session dies → login is then rejected forever ("A compositor is already active" from uwsm because the dead session's unit lingers) and the greeter itself crashes, so the only recovery is restarting the whole VM.

Frequency is escalating

DateWhat happened
10-054–5 RCU stalls (first observed)
10-07 18:15hard freeze → forced VM restart
10-07 22:51hard freeze → forced VM restart
10-07 22:53 → 10-08 08:56 (one 10h boot)17 RCU stall events (23:24–03:12), 10+ systemd watchdog kills, then a sudden hard freeze at 08:56 with no logged shutdown

Evidence from the journal

Block-layer / writeback hang (boot of 10-04 → 10-07):

10月 05 05:32:04 kernel: Workqueue: kblockd blk_mq_requeue_work
10月 05 05:32:04 kernel:  blk_mq_dispatch_rq_list+0x150/0x4cc
...
10月 06 23:47:28 kernel:  blk_mq_dispatch_queue_requests+0x1e0/0x208
10月 06 23:47:28 kernel:  blk_mq_flush_plug_list+0x90/0x17c
10月 06 23:47:28 kernel:  ext4_do_writepages+0x578/0x96c
10月 06 23:47:28 kernel:  ext4_writepages+0x88/0x118

Last night (boot 10-07 22:53 → 10-08 08:56) the signature shifted to outright starvation — a kworker blocked 160+ s and tasks stuck runnable:

10月 07 23:31:47 kernel: INFO: task kworker/6:2:28333 blocked for more than 160 seconds.
10月 07 23:31:47 kernel: task:pipewire-pulse  state:R  running task
10月 07 23:31:47 kernel:  unix_stream_connect+0x5e8/0x790
10月 07 23:59:48 kernel: rcu: Tasks blocked on level-0 rcu_node (CPUs 0-7): P35818 P588/1:b..l
10月 08 00:07:39 kernel: rcu: Tasks blocked on level-0 rcu_node (CPUs 0-7): P36997/1:b..l P504/1:b..l

and the watchdog cascade:

10月 07 23:29:52 systemd[1]: systemd-journald.service: Watchdog timeout (limit 3min)!
10月 07 23:43:59 systemd[1]: systemd-logind.service: Watchdog timeout (limit 3min)!
10月 07 23:59:07 systemd[1]: systemd-{userdbd,resolved,timesyncd,journald}.service: Watchdog timeout (limit 3min)!
10月 08 00:55:18 systemd[1]: systemd-resolved.service: Watchdog timeout (limit 3min)!

Ruled out on the guest side

  • No OOM (7.7 GiB RAM + zram, zero OOM records), no ext4 errors, no I/O errors on vda
  • File descriptors healthy (~5.5k open, limit far away) — the fd exhaustion from #326 is gone since the audio bridge was disabled
  • No failed systemd units; file system clean at 51% disk usage
  • Nothing user-triggered at freeze time (last log line before the 08:56 freeze is a completed fcitx5 autosave)

Questions

  1. The blk_mq_requeue_work / writeback stacks suggest virtio-blk requests are being requeued or not completed host-side. Is the VM disk a raw image on APFS? Would qcow2 / different cache settings change host-side latency behavior?
  2. Last night's stalls look more like vCPU starvation (tasks stuck state:R, idle CPUs reported "1 GPs behind") than disk hang — is there a known macOS host scheduling issue (e.g. under host memory pressure / efficiency cores)?
  3. What would be most useful to capture on the macOS side next time it freezes (QEMU args, spindump, log show)? Happy to collect.

Workaround for others hitting the "A compositor is already active" login lockout after such a freeze: don't bother trying to re-login, restart the VM directly.

Activity

1
yanUEd10d ago

Update (2026-10-12): another hard freeze — escalation continues.

New guest-side data since filing (boot 7410a41b, 2026-10-09 09:49 → 10-12 10:26 CST, ~3 days uptime, ended in a sudden hard freeze + manual restart).

Escalation table — this boot alone: 20 stall events + 17 systemd watchdog timeouts

daystall eventswindow
Oct 9 (evening)222:16–22:50
Oct 10800:49–13:27
Oct 11620:09–23:40
Oct 12402:09–08:48 (+ final hard freeze 10:26)

That beats the 17-stall night boot reported in the original issue.

Both stack signatures reproduced again

  1. Block-layer hang (Oct 11 23:40:53):
Workqueue: kblockd blk_mq_requeue_work
 blk_mq_dispatch_rq_list+0x150/0x4cc
 __blk_mq_sched_dispatch_requests+0xb0/0x1a0
 blk_mq_sched_dispatch_requests+0x2c/0x80
 blk_mq_run_hw_queue+0x288/0x2e0
  1. vCPU starvation (Oct 12 08:48): rcu_preempt kthread starved for 200119 jiffies!, task:rcu_preempt state:R running task, CPUs reported idle. This one cascaded into core services: systemd-logind and systemd-userdbd hit watchdog timeouts (limit 3min) → SIGABRT → SIGKILL at 08:46–08:48. The graphical session happened to survive; previous boots showed the session dying with logind.

Death mode this time

Sudden hard freeze at 10:26:25 CST — journal's last lines are mundane background activity (idle-monitor, xkbcomp keymap reload), zero shutdown/reboot entries; the VM was force-restarted ~2 minutes later. Same death mode as the two freezes on 10-07.

Ruled out on the guest side (same checklist as before)

  • 0 actual OOM kills (grep hits were RCU-warning text like "OOM is now expected behavior")
  • 0 EXT4-fs errors / I/O errors
  • 0 failed systemd units (system + user scope)
  • Disk 60% (14G/24G), ~4.5 GiB RAM available at restart
  • No FD exhaustion signature this time

Questions (unchanged from the issue body): especially — what should we capture on the macOS side at the next freeze (QEMU args, spindump, log show --last 1h)? I can harvest guest-side artifacts (full stall windows, blocked-task dumps) on the next occurrence on request.