#16107

Crash capture is silently disabled for the whole boot when plymouthd crashes on quit

Opendevzethopened 1d ago

System details

MacBookPro16,1 (Intel UHD 630 + AMD Radeon Pro 5500M, T2), Omarchy 4.5.0rc1, linux-t2 7.2.10, plymouth 26.134.222-3, systemd 262

What's wrong?

Summary

When plymouthd crashes while the boot splash is being closed, systemd-coredump switches core dump collection off for the rest of the boot. From then on no crash is recorded, coredumpctl stays empty, and omarchy-crash-watch never sends a "Process crashed" notification. The crash capture toggle still reports capture as enabled, so nothing tells the user that crash capture is off.

On the machine above, plymouthd crashes on every boot, so crash capture has been off in every session since install.

What happens

  1. plymouth-quit.service asks plymouthd to quit, and plymouthd segfaults while hiding the splash (cause below).
  2. plymouthd was started in the initramfs and survived switch-root (its argv[0] starts with @), so systemd keeps it in init.scope.
  3. systemd-coredump treats any crash in init.scope as a crash of PID 1. It logs "Due to PID 1 having crashed coredump collection will now be turned off" and writes |/bin/false to /proc/sys/kernel/core_pattern.
  4. Nothing restores the setting until the next reboot.

From the journal:

[   21.002629] systemd[1]: Starting Terminate Plymouth Boot Screen...
[   21.030924] systemd-coredump[1003]: Process 275 (plymouthd) of user 0 terminated abnormally with signal 11/SEGV, processing...
[   21.030928] systemd-coredump[1003]: Due to PID 1 having crashed coredump collection will now be turned off.
[   21.033171] systemd-coredump[1003]: Process 275 (plymouthd) of user 0 terminated abnormally without generating a coredump.
[   21.034709] systemd[1]: Finished Terminate Plymouth Boot Screen.

After login:

$ cat /proc/sys/kernel/core_pattern
|/bin/false

A later crash in the same boot, in this case boltd during resume from suspend, leaves only this behind:

kernel: coredump: 903(boltd): |/bin/false pipe failed

Why systemd does this

The behaviour comes from systemd commit c8091d9 (2016, "coredump: turn off coredump collection only when PID 1 crashes, not when journald crashes"). When PID 1 itself has crashed, socket activation and logging can no longer work, so processing further crashes could deadlock the machine. Turning collection off is a sensible safeguard for that case.

The check identifies PID 1 by its cgroup. In systemd 262 it is pidref.pid == 1 || streq_ptr(unit, SPECIAL_INIT_SCOPE) in src/coredump/coredump-context.c. init.scope also holds processes that are not PID 1: initramfs processes that survive switch-root, such as plymouthd, and the generators PID 1 runs. A crash in any of them switches collection off even though PID 1 is healthy.

Why plymouthd crashes

The core dump, symbolized with Arch's debuginfod:

#0  ply_list_get_first_node            ../plymouth/src/libply/ply-list.c:248
#1  ply_keyboard_remove_input_handler  ../plymouth/src/libply-splash-core/ply-keyboard.c:562
#2  stop_script_animation              ../plymouth/src/plugins/splash/script/plugin.c:397
#3  stop_animation                     ../plymouth/src/plugins/splash/script/plugin.c:422
#4  hide_splash_screen                 ../plymouth/src/plugins/splash/script/plugin.c:540
#5  ply_boot_splash_hide               ../plymouth/src/libply-splash-core/ply-boot-splash.c:653
#6  hide_splash                        ../plymouth/src/main.c:1269
#7  on_boot_splash_idle                ../plymouth/src/main.c:1370

It is a use-after-free. The ply_keyboard_t the script plugin still holds has already been freed. The list pointer read at frame 0 (0x56341d7b8) is the keyboard's own address (0x56341d7b8440) shifted right by 12 bits, which is glibc's safe-linking marker for a freed chunk.

In Plymouth's current main, free_keyboards_for_renderer() in ply-device-manager.c frees the keyboards of a renderer that goes away without notifying the splash. Pixel displays get a removed callback; keyboards do not, and the stored keyboard_removed_handler is never called. Here the splash starts on simpledrm, which goes away when the real GPU driver takes over, so the plugin is left with a dangling pointer and dereferences it when the splash is hidden. This applies to Omarchy's script-based theme and is likely not specific to this hardware.

I will report the Plymouth crash upstream on freedesktop GitLab and link it here. No existing upstream issue or merge request covers it. plymouth!513 fixes a different crash on quit.

Impact on Omarchy

  • omarchy-crash-watch follows the journal for systemd-coredump entries (MESSAGE_ID=fc2e22bc6ee647b6b90729ab34a250b1). With core_pattern set to |/bin/false, systemd-coredump is never invoked, so no entries appear and no notifications are sent.
  • coredumpctl records nothing, and the diagnose-crash workflow has no core to work with.
  • The state lasts until reboot and is invisible: the toggle shows capture enabled.

#10493 (a browser crash without a notification) might be related. It has not been diagnosed, and nothing confirms a link.

How to check

cat /proc/sys/kernel/core_pattern            # |/bin/false means collection is off
journalctl -b | grep 'PID 1 having crashed'  # shows what turned it off

Proposed fix

A pull request follows. It adds a oneshot service that re-applies the configured kernel.core_pattern after plymouth-quit.service. A real PID 1 crash freezes the system before the service could run, so the upstream safeguard keeps working in the case it was written for.

Activity

0

No activity yet.