#709

feat(audio): Add PCM audio over the audio codec offer

Opendraonccwants to merge intofrom

Summary

PCM becomes a fourth audio format, carried over the existing audio codec offer/accept mechanism. The default stays opus.

DESCRIBE
  codec_offer(audio_format, ac3_available, eac3_available)   # new, pure
    pcm  -> "PCM"                                            # no encoder needed
ANNOUNCE
  configure_codec(offered, accepted, ...)                    # was configure_dolby_audio
    offered == "PCM" && accepted == offered
      use_pcm = true
      packetDuration = 5 ms (client value ignored)
      HIGH_QUALITY, CUSTOM_SURROUND_PARAMS = off
      reply X-SS-Audio-Codec: PCM
    otherwise -> Opus audio, no error
  video bitrate -= pcm_bitrate_kbps(channels)                # 20% cap; PyroWave x1.5, uncapped
audio encode thread
  if use_pcm
    skip Opus/FFmpeg
    payload = pcm_payload(samples)      # *32768, round, clamp, int16 LE, interleaved
    raise packet; forward samples to WebRTC as the other paths do
 src/
 ├── audio.h/.cpp          # codec_e (+pcm), configure_codec, codec_name, codec_offer, pcm_bitrate_kbps
 ├── audio_policy.h/.cpp   # + pcm_payload
 ├── rtsp.cpp              # offer via codec_offer; PCM never refused; bitrate adjustment
 ├── config.cpp            # audio_format accepts "pcm"
 └── web UI + docs         # dropdown option, English strings, bandwidth notes

Other changes:

  • high_quality_audio is now visible only for ac3 and eac3, via a new oneOf visibility condition.
  • Docs and English strings describe PCM and its bandwidth (about 1.5, 4.6 and 6.1 Mbps for stereo, 5.1 and 7.1).

Evidence

These tests are added by the PR.

  • Before: PCM offered and accepted has no handling. configure_dolby_audio returns Opus, and audio_format = pcm is rejected by config validation. After (ANNOUNCE composition, 8 rows → 20):
    PCM/PCM stereo, 5.1, 7.1 (client sent 10 ms or none) -> 200, "PCM", use_pcm, 5 ms, Opus flags off
    PCM offered, nothing accepted                         -> "Opus"
    PCM offered, EAC3 accepted                            -> "Opus"
    AC3 offered, PCM accepted                             -> "Opus"
    bitrate: 2ch 97964, 6ch 94892, 8ch 93356 (of 100000)
    cap: 7.1 at 10000 kbps -> 7500
    PyroWave: 2ch 97196, 8ch 90284; 7.1 at 9000 kbps -> 400
    existing AC3/EAC3 rows unchanged
  • Before: no PCM conversion exists. After (AudioPcmPayload, no FFmpeg or hardware needed):
    pcm_payload({4660/32768, -4660/32768, 0}) == 34 12 CC ED 00 00   # little-endian, interleaved
    rounds half away from zero: 1.5 -> 2, -1.5 -> -2
    clamps 1.0, 1.5, 1e10 -> 0x7FFF; -1.0, -1.5, -1e10 -> 0x8000
    5 ms at 48 kHz -> 960 / 2880 / 3840 bytes for stereo / 5.1 / 7.1
  • After (UI): high_quality_audio.visibleWhen == { key: 'audio_format', oneOf: ['ac3','eac3'] }

Merge Danger

Door: two-way

PCM is opt-in. The audio format defaults to opus, so nothing changes until a user selects pcm. Reverting the PR leaves no persisted state behind. The one exception is a config that already says audio_format = pcm, which a revert would reject as an invalid value.

Blast Radius: negotiation

  • Shared negotiation code: the PR renames dolby_codec_e, configure_dolby_audio and dolby_codec_name, and moves the offer choice into codec_offer. The existing Dolby test rows are the guard against regressions. The branch is 5 commits behind master, so the rename may conflict with other audio work.
  • PCM sessions: PCM packets for 5.1 and 7.1 (2880 and 3840 B) rely on IP fragmentation. Losing any fragment loses the whole packet, so lossy networks get more silence gaps. PCM also reserves up to 6.1 Mbps from the video bitrate.
  • Unverified: only a PCM-capable client (Artemis) can exercise the PCM path. The audio encryption path is untouched and has no PCM-specific test.

Problem Statement

Vibepollo sends audio to Artemis as Opus audio, or as Dolby audio (AC-3 / E-AC-3) when the audio format selects it and the client accepts it. Both are lossy, so music, soundtracks and positional surround cues arrive with compression artifacts even on networks with spare bandwidth. Dolby audio also only covers stereo and 5.1, so there is no way to get the host's audio bit-for-bit in any channel layout, 7.1 included.

Solution

Make PCM audio a fourth choice of the audio format (pcm), carried over the existing codec offer/accept mechanism. With the audio format on PCM, DESCRIBE offers PCM. A client that accepts it in ANNOUNCE gets 16-bit signed, 48 kHz audio in stereo, 5.1 or 7.1 in 5 ms packets. A client that doesn't accept it silently gets Opus audio, as today. The default stays opus, and Dolby behaviour is unchanged.

DESCRIBE   a=x-ss-audio-codec-offer:PCM      (audio format = pcm)
ANNOUNCE   x-ss-audio-codec-accept:PCM
reply      X-SS-Audio-Codec: PCM
audio      encode thread skips Opus/FFmpeg -> int16 LE payload, 5 ms packets

Implementation Decisions

  • Config / UI / docs: audio_format accepts pcm (config.cpp). The web UI dropdown gets a PCM option with English strings only (settingsSchema.ts, en.json). docs/configuration.md documents PCM audio, its per-layout bandwidth (about 1.5 / 4.6 / 6.1 Mbps) and the high quality audio note.
  • Offer choice extracted: audio::codec_offer(format, ac3_available, eac3_available) is a pure function used by DESCRIBE. pcm always offers PCM because no encoder is needed.
  • Dolby-specific names generalized: dolby_codec_e / configure_dolby_audio / dolby_codec_name become codec_e (opus, ac3, eac3, pcm) / configure_codec / codec_name. config_t gains use_pcm beside use_ac3 / use_eac3, which keeps the diff against upstream small.
  • Negotiation: PCM is selected only on an exact offer/accept match. The packet duration is fixed to PCM_PACKET_DURATION (5 ms) regardless of the client's x-nv-aqos.packetDuration. The HIGH_QUALITY and CUSTOM_SURROUND_PARAMS flags are turned off, and the high quality audio setting is ignored. ANNOUNCE can only refuse (400) a Dolby accept the host can't provide. PCM supports every channel layout, so it is never refused for that reason.
  • Capture path: when use_pcm is set, the encode thread skips the Opus and FFmpeg encoders. policy::pcm_payload scales each float sample by 32768, rounds to nearest, clamps, with no dither, and writes little-endian int16 in interleaved order (audio_policy.cpp). Samples are forwarded to WebRTC under the same condition as the other paths.
  • Bitrate budget: audio::pcm_bitrate_kbps (48000 × 16 × channels = 1536 / 4608 / 6144 kbps) feeds the existing video bitrate adjustment, and the existing 20% cap and PyroWave rules (× 1.5, uncapped, session refused if it doesn't fit) apply unchanged.
  • ANNOUNCE reply: X-SS-Audio-Codec reports PCM via codec_name.
  • Unchanged: the 4096-byte audio packet limit, audio encryption, and 4+2 Reed-Solomon recovery.
  • Related UI fix (not in the original issue): high_quality_audio is now shown only for ac3 / eac3, using a new oneOf condition in the settings schema and SettingsView.vue, since the setting has no effect for Opus or PCM.

Testing Decisions

  • ANNOUNCE composition test (test_host_stream_terminal_composition.py, composition.cpp.in), which compiles the real ANNOUNCE handler with the real negotiation code. The fixture now follows the renamed functions, includes codec_offer and pcm_bitrate_kbps, and records the adjusted video bitrate (allocated_bitrate). The table grows from 8 to 20 rows, and the existing Dolby rows keep passing and gain bitrate expectations.
    • PCM accepted in stereo / 5.1 / 7.1 gives PCM, 5 ms (client-sent 10 ms or none), and Opus-only flags off.
    • PCM offered but not accepted, PCM offer with EAC3 accepted, or AC3 offer with PCM accepted all give Opus.
    • The high quality audio setting doesn't change a PCM session.
    • The bitrate reduction is checked as capped at 20% (7500 of 10000 kbps at 7.1), and PyroWave is checked as reserved in full (× 1.5) or refused with a 400 at 9000 kbps.
    • codec_offer is checked for opus, ac3, eac3 and pcm, and pcm_bitrate_kbps for 1536 / 4608 / 6144.
  • Audio policy unit tests (tests/unit/test_audio.cpp) cover pcm_payload: little-endian int16 in interleaved order, round-half-away-from-zero, clamping of full-scale and out-of-range values, and 960 / 2880 / 3840 B for 5 ms of stereo / 5.1 / 7.1. These need no audio hardware or FFmpeg.
  • UI (settings-behavior.test.ts): high_quality_audio is visible only for ac3 / eac3.
  • Tests were not run as part of writing this description.

User Stories

Coverage of the Vibepollo user stories from draoncc/Vibepollo-Old#11, checked against this diff. Client-side and Apollo stories live in other repositories and are out of scope for this PR.

#StoryIn this diff
1PCM is a choice of the audio format✅ config.cpp, settingsSchema.ts
2Audio format still defaults to Opus✅ default unchanged
3Docs state PCM bandwidth per layout✅ configuration.md, en.json
4Docs say PCM supports 7.1, unlike Dolby audio✅ configuration.md
5PCM appears in the web UI dropdown✅ settingsSchema.ts, en.json
6Clients that don't accept PCM get Opus, no error✅ configure_codec, tested
7Log line when the client didn't accept PCM⚠️ not in this diff
8Dolby audio behaves as today✅ existing Dolby rows keep passing
9Video bitrate lowered for PCM, capped at 20%✅ rtsp.cpp, tested
10PyroWave counts PCM + recovery in full✅ via existing branch, tested
11PyroWave refuses a PCM session that can't fit✅ tested (400 at 9000 kbps)
12PCM packets encrypted like Opus/Dolby✅ encryption path untouched; no PCM-specific test
13ANNOUNCE reply reports PCM✅ codec_name
14Host fixes PCM packets at 5 ms✅ PCM_PACKET_DURATION, tested
15Opus quality flag, surround params, high quality setting have no effect✅ configure_codec, tested
16WebRTC audio unaffected✅ PCM path forwards samples under the same condition as the Opus and Dolby paths
34PCM covered by the ANNOUNCE composition test✅
35Offer choice covered by tests✅ codec_offer
36Sample conversion tested without hardware/FFmpeg✅ test_audio.cpp
37Dolby names generalized✅
38Change kept small against upstream✅ use_pcm flag beside use_ac3 / use_eac3

Stories 17–33 and 40 (Artemis, Apollo, moonlight-common-c) are not part of this PR.