Description

Sunshine randomly crashes when a second client connects while another one is already streaming. It doesn't happen on every connect (about one in three in my case), but when it does, every client's stream drops at once. Seen on Linux with NVENC and wlr capture (headless labwc, NVIDIA 615.x). The only trace is in the kernel log:

kernel: session::video[...]: segfault at 0 ip 0000000000000000 sp ... error 14

The cause is that cuda::init() reloads the shared CUDA function table (cdf) on every call, and every new stream calls it when it creates its encode device. cuda_load_functions() frees the current table, installs a zeroed one, and only then opens libcuda.so.1 and fills the entries in. A stream that is already encoding calls through cdf on every frame. If it does so inside that window, it calls a null function pointer and the whole process dies.

Backtrace from the core dump (v2026.914.233613; master has the same init()):

Thread 1 (the stream already running)
#0  0x0000000000000000 in ?? ()
#1  cuda::gl_cuda_vram_t::convert(platf::img_t&) ()      <- call *0x278(%rax), rax = freshly zeroed table
#2  stream::videoThread(stream::session_t*) ()

Thread 15 (the stream that just started)
#1  _dl_lookup_map (name="libcuda.so.1")
#7  dlopen
#8  cuda::init() ()
#9  cuda::make_avcodec_gl_encode_device(int, int, int, int) ()
#10 wl::wlr_vram_t::make_avcodec_encode_device(platf::pix_fmt_e) ()
#11 video::make_encode_device(...) ()
#12 stream::videoThread(stream::session_t*) ()

This change loads the table once and keeps it for the life of the process. Later calls only run cuInit(0) again, so a first load that failed is still retried. A mutex stops streams that start at the same time from loading it concurrently. To test this without a GPU, cuda::init() now hands the table and the loader to an init(table, load) overload, the same way device_pci_bus_id() takes its driver operations.

Screenshot

Issues Fixed or Closed

Roadmap Issues

Testing

I first made the crash reproducible on demand, so it happened on every attempt instead of about one connect in three. Then I applied the fix, rebuilt Sunshine, and ran it on the same machine. The end-to-end tests with the fix were clean: no crashes, and both streams kept running. The crash happened every time before the fix and never after it, which confirms the fix resolves it. The details are below.

Regression tests in tests/unit/platform/linux/test_cuda.cpp. They use a fake loader, so they need no GPU and run in CI:

CudaInitTest.LoadsTheFunctionTableOnlyOnce: a second init() must not load the table again. It fails on the first commit (the table is loaded twice) and passes with the fix, in 20 out of 20 repeats each.
CudaInitTest.RetriesLoadingAfterAFailure: a load that failed is retried on the next call. It passes on both commits.

Full test_sunshine suite, in a Fedora 44 container: 862 passed and 1 failed (the test above) on the first commit, and 863 passed with the fix. 14 tests were skipped in both runs.

Against the real driver (Fedora 44, RTX 3050, driver 615.71.10): a manual check, not part of this PR, intercepted dlopen("libcuda.so.1") during a second init(). Without the fix, the entry every frame conversion calls first (cuGraphicsMapResources) was null in 30 out of 30 runs.

End to end: headless labwc, wlr capture, NVENC, with one client streaming continuously. A preloaded shim delayed dlopen("libcuda.so.1") by 100 ms to widen the window:

without the fix: a second client connected 3 times, and all 3 crashed Sunshine with the segfault above
with this PR: a second client connected 7 times, with no crashes, and both streams kept running

Built with BUILD_WERROR=ON (gcc 15, CUDA 13.1.1). clang-format reports no changes. Not run: CI, Windows and macOS (the change is Linux-only).

Type of Change

Checklist

Code follows the style guidelines of this project
Code has been self-reviewed
Code has been commented, particularly in hard-to-understand areas
Code docstring/documentation-blocks for new or existing methods/components have been added or updated
Tests have been added or updated for new or modified functionality, including regression tests for bug fixes