KWin + VA-API: Black screen on RX 6700 XT with EGL_BAD_PARAMETER (0x300C); memory-buffer workaround causes high latency

Is there an existing issue for this?

• I have searched the existing issues

Closely related to #5316, but with different GPU hardware and symptoms (black screen instead of a crash).

Is your issue described in the documentation?

• I have read the documentation

Is your issue present in the latest beta/pre-release?

This issue is present in the latest pre-release

Describe the Bug

Sunshine 2026.914.233613 produces a completely black video stream when using KWin/PipeWire capture with VA-API hardware encoding on an AMD Radeon RX 6700 XT (Navi 22) passed through to a Debian 13 VM running KDE Plasma on Wayland.

Moonlight successfully connects and audio works, but no video is displayed. The issue occurs on both Windows 11 and Debian Linux Moonlight clients over wired LAN, regardless of whether Desktop or Steam Big Picture is streamed.

Sunshine repeatedly reports:

Info: Creating encoder [hevc_vaapi]
Error: Couldn't import RGB Image: 0000300C
Error: Could not convert image

0x300C corresponds to EGL_BAD_PARAMETER.

Configuration:

capture = kwin
encoder = vaapi
adapter_name = /dev/dri/renderD128
output_name = Virtual-2

The RX 6700 XT is correctly detected, and vainfo confirms H.264 and HEVC hardware encoding support.

Test results:

Explicit GPU selection using adapter_name and DRI_PRIME=pci-0000_01_00_0 did not resolve the issue.

Workaround confirmed:

Following the approach described in #5316, I rebuilt Sunshine from the same version and modified src/platform/linux/pipewire.cpp to force CPU-memory buffers by setting:

n_dmabuf_infos = 0;

The patched build reports:

[pipewire] using memory buffers
Found H.264 encoder: h264_vaapi [vaapi]
Found HEVC encoder: hevc_vaapi [vaapi]

The video stream now works with VA-API hardware encoding enabled.

However, the resulting stream has noticeably higher latency and feels sluggish, even over wired LAN. The additional memory copies may contribute to this, particularly at 3840x2160. This has not yet been fully benchmarked.

Unlike the original report in #5316, Sunshine does not crash. Instead, it repeatedly fails to import the captured image while continuing to run.

Expected Behavior

KWin/PipeWire capture should successfully import frames into the VA-API hardware encoding pipeline without producing a black screen.

Ideally, Sunshine should support low-latency DMA-BUF capture on this configuration or gracefully fall back to memory buffers when DMA-BUF/EGL import fails.

Hardware encoding should remain responsive at 4K/60 FPS without requiring an experimental patch.

Additional Context

• Related issue: #5316
• The original issue involves an AMD Polaris GPU, while this reproduction uses RDNA2 (RX 6700 XT).
• Sunshine runs in a Proxmox VM with PCIe passthrough of the Radeon GPU.
• The VM additionally has a QEMU/Bochs display adapter (1234:1111) for its graphical console.
• KDE Plasma runs on Wayland with a virtual KWin display (Virtual-2, 3840x2160 @ 60 Hz, 200% scaling).
• A second virtual display (Virtual-1) was disabled without resolving the problem.
• Capturing a normal desktop screenshot with KDE Spectacle works correctly.
• The patched Sunshine version was built locally and started through a systemd user-service override.
• Both Windows 11 and Debian Linux Moonlight clients reproduce the black screen.
• The underlying cause appears related to DMA-BUF/EGL image import between KWin/PipeWire and Sunshine's VA-API processing, but this has not been conclusively isolated.

Host Operating System

Linux

Operating System Version

Debian GNU/Linux 13 (Trixie), KDE Plasma / Wayland, kernel 6.12.113+deb13-amd64, running inside a Proxmox VM

Architecture

amd64/x86_64

Package

Linux - AppImage

GPU Type

AMD

GPU Model

AMD Radeon RX 6700 XT (Navi 22, RDNA2, PCI ID 1002:73df), PCIe passthrough

GPU Driver/Mesa Version

amdgpu, Mesa 25.0.7-2+deb13u1, radeonsi, LLVM 19.1.7, DRM 3.61

Capture Method

Kwin

Apps

[
  {
    "name": "Desktop"
  },
  {
    "name": "Steam Big Picture"
  }
]

Log output

[2026-10-28 20:00:12.316]: Info: Creating encoder [hevc_vaapi]
[2026-10-28 20:00:12.329]: Info: EGL: context priority set to HIGH
[2026-10-28 20:00:12.336]: Info: vaapi vendor: Mesa Gallium driver 25.0.7-2+deb13u1 for AMD Radeon RX 6700 XT (radeonsi, navi22, LLVM 19.1.7, DRM 3.61, 6.12.113+deb13-amd64)
[2026-10-28 20:00:12.336]: Info: Streaming bitrate is 14988000
[2026-10-28 20:00:12.336]: Info: [VAAPI] Quality level set to 0 (fastest level: 32)
[2026-10-28 20:00:12.336]: Info: [VAAPI] Using CBR rate control with standard VBV size
[2026-10-28 20:00:12.369]: Error: Couldn't import RGB Image: 0000300C
[2026-10-28 20:00:12.369]: Error: Could not convert image
[2026-10-28 20:00:12.377]: Info: Creating encoder [hevc_vaapi]
[2026-10-28 20:00:12.389]: Info: EGL: context priority set to HIGH
[2026-10-28 20:00:12.396]: Info: vaapi vendor: Mesa Gallium driver 25.0.7-2+deb13u1 for AMD Radeon RX 6700 XT (radeonsi, navi22, LLVM 19.1.7, DRM 3.61, 6.12.113+deb13-amd64)
[2026-10-28 20:00:12.426]: Error: Couldn't import RGB Image: 0000300C
[2026-10-28 20:00:12.426]: Error: Could not convert image

After applying the memory-buffer workaround:

[pipewire] Streaming display 'Virtual-2' offset: 0x0 resolution: 3840x2160
[pipewire] using memory buffers
Found H.264 encoder: h264_vaapi [vaapi]
Found HEVC encoder: hevc_vaapi [vaapi]

Online logs

No response