Linux AMD RDNA4 (RX 9070 XT): all VA-API encoders fail first init ("No usable encoding entrypoint"), HDR colors wrong after fallback

Bug symptoms

On an AMD RDNA 4 host (RX 9070 XT, Mesa radeonsi VA-API), HDR streams have badly wrong colors, while the same stream in SDR looks correct.

The log shows that every VA-API encoder fails its first initialization with No usable encoding entrypoint found, although vainfo lists VAEntrypointEncSlice for all of these profiles. Vibepollo then retries with fallback options and the stream starts:

Error: [h264_vaapi] No usable encoding entrypoint found for profile VAProfileH264High (7).
Error: [hevc_vaapi] No usable encoding entrypoint found for profile VAProfileHEVCMain (17).
Error: [hevc_vaapi] No usable encoding entrypoint found for profile VAProfileHEVCMain10 (18).
Error: [av1_vaapi]  No usable encoding entrypoint found for profile VAProfileAV1Profile0 (32).
Info: Retrying with fallback configuration options for [hevc_vaapi] after error: Function not implemented

For SDR this is invisible because the fallback works. For HDR (HEVC Main10) the stream is still signalled as HDR (Rec. 2020 + SMPTE 2084 PQ) but the colors on the client are wrong. My guess, not confirmed: the fallback path does not keep the 10-bit / P010 configuration.

HDR session log excerpt:

Info: Linux private display [1]: KScreen operation=HDR activation result=succeeded.
Info: Using event-driven KMS capture for Vibepollo DRM CRTC [76].
Info: Client dynamicRange: 1, Display is HDR: true
Info: Creating encoder [hevc_vaapi]
Info: Color coding: HDR (Rec. 2020 + SMPTE 2084 PQ)
Info: Applying 256-entry CRTC GAMMA_LUT during KMS capture.
Error: [hevc_vaapi] No usable encoding entrypoint found for profile VAProfileHEVCMain10 (18).
Info: Retrying with fallback configuration options for [hevc_vaapi] after error: Function not implemented
Info: Linux encoder dimensions: encoder=vaapi capture=3840x2160 encoder_context=3840x2160 stream_fps=120 ... result=ready.

With AV1 there is also: Warning: [av1_vaapi] Multiple slices were requested but this codec does not support controlling slices.

A similar mismatch (vainfo lists the Main10 encode entrypoint, ffmpeg hevc_vaapi reports no usable entrypoint) was reported in HandBrake: https://github.com/HandBrake/HandBrake/issues/8043

Additional observation: host processing latency at 4K120 (SDR)

Moonlight statistics, managed virtual display 3840x2160@120, wired 2.5 GbE LAN (1 ms, 0 % loss), fullscreen video playing:

Client decode time is 3.4–4.7 ms, so the bottleneck is on the host. The KWin bridge matches the installed KWin (libkwin.so.6.8.2) and the logs show no CPU-upload fallback, so this does not look like #657. It may be related to the encoder running with the fallback options.

Expected behavior

• The VA-API encoders initialize with the primary options on RDNA 4, since vainfo reports encode support for H.264, HEVC Main/Main10 and AV1.
• HDR streams over HEVC Main10 (and AV1) have correct colors.

Steps to reproduce

1. Install Vibepollo 2.0.6 on CachyOS, KDE Plasma 6.8.2 Wayland, AMD RX 9070 XT.
2. Pair Moonlight (macOS, MacBook Air M5 driving an external 4K HDR monitor) and enable HDR.
3. Start a Desktop stream with the managed virtual display at 3840x2160@120.
4. Colors are wrong. Disable HDR in Moonlight and the colors are correct.

Environment

• Vibepollo: 2.0.6-1 (release package, sha256 verified)
• OS: CachyOS, kernel 7.3.1-1-cachyos
• Desktop: KDE Plasma / KWin 6.8.2-1.1, Wayland, SDDM
• GPU: AMD Radeon RX 9070 XT (Navi 48, gfx1201) [1002:7550]
• Mesa: 26.2.5 (radeonsi), libva 2.24.1
• Virtual DRM: vibeshine_drm 1.5.0 (registers as /dev/dri/card0, amdgpu is card1, render node renderD128)
• Client: Moonlight on macOS, MacBook Air M5, wired 2.5 GbE LAN, output to an external 4K 240 Hz HDR QD-OLED (MSI MPG 321URX), so 4K120 HDR is the actual target

vainfo (run as the desktop user):

vainfo: Driver version: Mesa Gallium driver 26.2.5-arch3.1 for AMD Radeon RX 9070 XT (radeonsi, gfx1201, ACO, DRM 3.64, 7.3.1-1-cachyos)
      VAProfileH264ConstrainedBaseline:	VAEntrypointEncSlice
      VAProfileH264Main               :	VAEntrypointEncSlice
      VAProfileH264High               :	VAEntrypointEncSlice
      VAProfileHEVCMain               :	VAEntrypointEncSlice
      VAProfileHEVCMain10             :	VAEntrypointEncSlice
      VAProfileAV1Profile0            :	VAEntrypointEncSlice

I'm happy to run debug builds or provide full logs.

Disclaimer: This report was written with the help of Claude (Anthropic AI assistant). The logs, vainfo output and Moonlight measurements were collected on my system.