perf: constant QP with voxl's own rate control (~30% less CPU, lower latency)

What

x264 now encodes at a constant QP chosen per frame by voxl, with uneven multi-hexagon (UMH) motion search. voxl does its own rate control instead of x264's CRF + VBV. The result is 26–37% less voxl CPU on every workload and link, lower latency, and a sharper picture at the same or lower bitrate. On the benchmark nothing got worse: a few cases are level, within run-to-run noise.

The real web client in Chromium, mid-scroll: 59.0 vs 60.0 fps, 3.29 vs 3.34 Mbps, server pipeline 7.9 → 5.5 ms, same text sharpness. The black squares top left are the benchmark's latency marker.

Full report with charts and every metric: voxl encoder benchmark

Why

CRF and VBV need x264's lowres lookahead: a half-resolution copy of every frame plus a motion search over all of it, mostly before the sliced threads can start. In an instrumented x264 build (4 sliced threads, a 1080p region changing in a 2560x1600 frame), a frame took 7.4 ms with CRF + VBV and 4.5 ms at constant QP. Macroblock encoding costs the same in both modes, so the lookahead is pure overhead.

#22 tried constant QP and found nothing measurable, because without the lookahead's motion vectors ultrafast's diamond search loses scrolls (56% bigger frames on recorded Chromium scrolling). UMH finds them on its own: 9.1 KB vs CRF's 12.2 KB per frame at +1.4 dB, in 3.8 ms instead of 7.5 ms. Constant QP also needs rate control, which voxl now does itself. It already knows what changed on screen and how fast each viewer's link is.

Results

voxl on a 16 vCPU VM, native client, interleaved A/B runs (3 per case, 7 where runs disagreed). Shaped links use bench/shaper.py. Capped runs are a fast link with --max-kbps=6000. Quality was re-measured after the PSNR fix below (2–3 runs per shaped or capped case). After the review fixes, the full matrix plus 60-second sessions on the slow links was re-run against the original release, with the same conclusions: lower CPU, encode time, p50 and p99 latency in every case.

• CPU: encode time per frame is down 34–44%. Xvfb's CPU is unchanged.
• FPS: 60 everywhere, as before.
• Quality: measured with the new VOXL_PSNR=1 knob (x264's own measurement of what viewers decode). It's the same or better at the same or lower bitrate, except 1080p on 10 Mbps. There the new controller uses headroom the link has (measured at 9.4 Mbps) for 9 dB more, and its latency is still lower.
• Level rather than ahead, within run-to-run noise:
  • Scroll on 5 Mbps: p99 118 → 119 ms, worst second 59.1 → 58.4 fps.
  • Small update's worst second: 59.9 → 59.5 fps, one dropped frame in one of three runs.
  • Full screen under the 6 Mbps cap: p99 20 → 21 ms.

Known issue: frame pacing at 5 Mbps

The table's metrics count frames per second, so they miss this. On a 5 Mbps link, frames arrive less evenly than before: gaps of more than 33 ms between frames (a brief hitch) are about 30 vs 3.5 per 22 s on 1080p animation, and about 6–10 vs 1.5 on scroll. At 10 Mbps it's about 8 vs 5, and on LAN there are none in either build. The cause is QP oscillation. A frame that steps QP down by one re-sends the detail its reference lost and comes out 2–3× bigger, which takes 25–40 ms on a 5 Mbps link. x264's CRF kept QP steadier. Per-frame size caps, growth limits and re-encoding big frames didn't fix it. Not fixed in this PR; the likely fixes are spreading each step-down over several frames (per-macroblock QP offsets) or holding QP steadier on slow links.

How rate control works now

• Per-frame QP (QpCtl in src/video.rs): byte models per frame kind (motion, refine, keyframe) predict each frame's size from its changed macroblocks. A VBV-style buffer refills at the cap, as x264's did, and voxl picks the sharpest QP that fits. Updates under a quarter of the screen ask for up to 6 QP sharper (typing, spinners), and keyframes ask for 3 sharper.
• Re-encode oversized frames, keep the stream valid (src/h264.rs): a frame far over its budget is invalidated in x264 (x264_encoder_invalidate_reference, with one spare reference that costs no search time) and re-encoded from the frame before. In its place goes the dropped frame's own slice headers with every macroblock skipped, so frame_num stays continuous. This placeholder decodes bit-exact against x264's reconstruction in FFmpeg. More than 700 re-encodes in Chromium's WebCodecs gave no stalls or decoder errors. Re-encoding without a placeholder had stalled Chromium in 1 of 4 runs.
• Measure the link (src/hub.rs): each viewer's capacity comes from large frames' acknowledgement delay over the ping round trip. The bitrate target stays under 80% of it, and the frame buffer never refills faster than 90% of it. Before, an idle link let the bitrate controller ramp to twice what the content needed, so bursts after a scene change queued up. An estimate no frame has confirmed for 20 s is dropped if it could be holding the rate down (below --max-kbps / 0.8), because frames held small by it can't get large enough to measure the link again.
• Recover at the right speed: a frame sharper than its reference also re-encodes the detail the reference lost. voxl learns that cost per QP step, and takes several steps per frame only on links measured at over twice the cap with plenty of room. Otherwise it takes one step at a time.
• --max-kbps is respected on average: bursts may exceed it, as before, and a slow outer loop raises the QP while the stream's sent rate (all stripes together) stays above it.

Other changes

• Conversion: persistent conversion threads replace per-frame scoped threads, and only the changed columns are converted (src/convert.rs, src/capture.rs, src/win/capture.rs). macOS ignores the columns.
• /stats: now reports the average QP and each viewer's measured capacity.
• bench/perfab.py: interleaved A/B runs with FPS, latency, CPU and a per-thread breakdown in one run. nclient.py can log every frame with LATLOG=path.

Changes since review

All three review comments were real bugs:

• Stale capacity estimates: a low estimate could hold quality down indefinitely. In a test where the link stepped from 5 to 50 Mbps, scroll stayed at QP 28–30 for the rest of the session; now it returns to QP 20 once the estimate goes stale. Expiring every estimate after 20 s, as suggested, also forgot that fast links are fast, and slowed LAN scene-change recovery from 69 to 86 frames on average. So only an estimate that could limit the rate is dropped (72 frames, level).
• --max-kbps per stripe: with --stripes=4, each stripe was held to a quarter of the cap even when the others were idle. It's now measured over the whole stream: 2 of 4 stripes moving under a 6 Mbps cap now send 3.69 instead of 2.95 Mbps, at QP 30.8 instead of 32.2.
• PSNR of discarded frames: VOXL_PSNR counted frames that were re-encoded and never sent. It now counts only frames viewers get, and the quality column above was re-measured.

Merged main (Windows support); its capture backend now takes the changed columns too.

Behaviour changes

• New flag --qp, default 20: it replaces --crf (default 24). --crf N still works and is read as QP N.
• --paintover-qp: replaces --paintover-crf, which still works.
• Placeholders need CAVLC: the default --preset ultrafast uses it. Other presets fall back to swapping in a keyframe for an oversized frame.

Testing

• cargo test: 7 pass, including new Exp-Golomb, emulation-prevention, placeholder, capacity-staleness and stream-wide --max-kbps tests.
• rustfmt, plus clippy on Rust 1.86 and 1.96 with no new lints.
• macOS (aarch64-apple-darwin) type-check.
• Dynamic-resolution end-to-end test (bench/tests/resizedyn.mjs): 8/8 on the final binary.
• --stripes=4 and --adaptive-threads runs.
• The real web client in Chromium through bitrate-capped scene changes: 0 stalls, 0 decoder errors.
• Not tested: Safari and Firefox decoders (not available here) and real desktop content beyond the synthetic workloads and the Chromium scroll page.

Reproduce:

cd bench && OUT=/tmp/ab.jsonl python3 perfab.py --reps 3 \
  --workloads anim1080,animfull,scroll,animsmall --net lan,10/40,5/60 \
  base=/path/to/old/voxl new=../target/release/voxl

Want tembo to make any changes? Add a comment with @tembo and i'll get back to work!