Description

At full color range with 4:2:0, the CUDA NV12 kernel (RGBA_to_NV12 in src/platform/linux/cuda.cu) writes incorrect chroma for pure red and pure blue. Moonlight then shows #ff0000 as green and #0000ff as dark green. This code runs on Linux with an NVIDIA GPU.

The cause is an unclamped store. The kernel scales chroma by 256 and stores the float in a uint8_t. For #ff0000, V is 128/255 + 0.5, approximately 1.002. Scaled, this gives 256.5, and the store wraps it to 0. Blue has the same problem with U. #fe0000 and #0000fe give approximately 256.0, so the rounding of the float calculation decides if they wrap. The 4:4:4 kernel scales by 255, has a maximum of 255.5 and cannot wrap. I did not change it.

Changes:

src/platform/linux/cuda_color.h is a new header. Its two functions compile for the GPU and for the host:
- clamp_to_uint8() clamps a sample to the range 0 to 255. NaN gives 0.
- nv12_chroma_sample() calculates the average of a 2x2 block in the same order as the old kernel code. Then it clamps the result.
RGBA_to_NV12 uses nv12_chroma_sample(). The arithmetic is the same as before, except for the clamp.
cuda::test::convert_to_nv12() is a test hook in cuda.cu and cuda.h. It runs the real NV12 kernel on a frame from host memory. Only builds with SUNSHINE_TESTS include it.

I put the clamp in a header so that unit tests can check it on a host without a GPU.

Screenshot

Before

After

Issues Fixed or Closed

Fixes #5953

Roadmap Issues

Testing

I added tests/unit/platform/linux/test_cuda_color.cpp. It has two test suites:

CudaColorTest (5 tests) runs on all hosts. It checks these items:
- The clamp at the edges, at ±infinity and at NaN.
- Full-range V of #ff0000 and U of #0000ff stay at 255 for Rec. 601, Rec. 709 and Rec. 2020.
- 11 colors in all six 8-bit colorspaces agree with the ITU-T H.273 formula, within 1 code value.
- Limited-range chroma stays between 16 and 240.
- The chroma of a block is the average of its four pixels.
CudaNv12KernelTest (2 tests) runs the real kernel through the test hook on solid frames. It checks the chroma plane against the same formula. It also checks that full-range red and blue keep the maximum value. The tests skip if there is no CUDA device, as on CI.

Checks that I ran:

I built CudaColorTest with GoogleTest and Apple clang (C++23) on macOS. I used a copy of color_vectors_from_colorspace() from src/video_colorspace.cpp. All 5 tests pass.
Regression check: I put the old unclamped store back into nv12_chroma_sample(). Then FullRangePureRedAndBlueKeepMaximumChroma and Nv12ChromaMatchesTheColorspaceMatrix fail, with V of red and U of blue at 0.
CudaNv12KernelTest compiles against src/platform/linux/cuda.h (clang -fsyntax-only).
clang-format with the repository configuration makes no changes to the changed files.
Manual test: I ran Sunshine with this change on the same host and client as in #5953 (Moonlight for macOS, HEVC 8-bit, 4:2:0, full range). I used the six-color test card from the issue. Before the fix, #ff0000 shows as green and #0000ff shows as dark green. With the fix, all six colors show correctly. I added screenshots from before and after the fix.

Type of Change

feat: New feature (non-breaking change which adds functionality)
fix: Bug fix (non-breaking change which fixes an issue)
docs: Documentation only changes
style: Changes that do not affect the meaning of the code (white-space, formatting, missing semicolons, etc.)
refactor: Code change that neither fixes a bug nor adds a feature
perf: Code change that improves performance
test: Adding missing tests or correcting existing tests
build: Changes that affect the build system or external dependencies
ci: Changes to CI configuration files and scripts
chore: Other changes that don't modify src or test files
revert: Reverts a previous commit
BREAKING CHANGE: Introduces a breaking change (can be combined with any type above)

Checklist

Code follows the style guidelines of this project
Code has been self-reviewed
Code has been commented, particularly in hard-to-understand areas
Code docstring/documentation-blocks for new or existing methods/components have been added or updated
Tests have been added or updated for new or modified functionality, including regression tests for bug fixes

AI Usage

See our AI usage policy.

AI tools were used while preparing this PR

I used an AI agent. It found the cause in cuda.cu, wrote the fix and the tests, ran the checks above, and wrote a draft of this description. I found the bug, reproduced it with a test card, and tested the workarounds in #5953.