feat(mac): hardware H.264 encoding with VideoToolbox

What

--encoder=videotoolbox encodes H.264 on the Mac's media engine through VideoToolbox, instead of x264 on the CPU. It's opt-in: x264 stays the default until this has been measured on real Mac hardware (see "Not measured yet"). A Mac without a usable hardware encoder falls back to x264 with one warning. On Linux and Windows the flag warns and uses x264.

How

• src/mac/vt.rs has the same interface as the x264 wrapper: I420 in, Annex B out, a keyframe when asked for.
  • Each frame is copied into an NV12 pixel buffer from the session's pool.
  • It's encoded in low-latency mode (EnableLowLatencyRateControl, real time, no frame reordering, hardware required).
  • encode waits for its frame, matched by number, so a late callback for a dropped frame can't be mistaken for it.
• VideoToolbox writes AVCC, so the output is rewritten as Annex B, with SPS and PPS in front of every keyframe the way x264 writes them. Viewers, recordings and the web client take it unchanged.
• The VUI color tags match x264's (BT.709 primaries and transfer, BT.601 matrix, limited range).
• Rate control maps onto voxl's:
  • The bitrate cap becomes AverageBitRate.
  • The CRF becomes MinAllowedFrameQP, so the stream doesn't spend more than x264 would, and paint-overs (a lower CRF) still sharpen still content. That property is looked up at run time, because older macOS versions don't have it.
  • Keyframes get the same budget boost through set_rate_buf.
  • x264's macroblock hints have no VideoToolbox equivalent, and the adaptive-threads governor is skipped for a hardware encoder.
• Encoder in src/encoder.rs is now an enum (x264, or VideoToolbox on macOS). video.rs passes the choice through EncoderConfig::hardware.
• Every VideoToolbox symbol linked directly dates from macOS 11.3 or earlier. voxl already needs 12.3 for ScreenCaptureKit, so this doesn't raise the minimum. VideoToolbox is a system framework, so the Tembo CLI's macOS linkage check still passes.
• CI's macOS job now runs cargo test too, which encodes through VideoToolbox on the runner.

Results (CI's macOS 26 runner, a virtual Apple Silicon Mac)

From vt::tests::encode_cost (ignored by default; run it with cargo test --release vt::tests::encode_cost -- --ignored --nocapture). The content is a 2560x1600 desktop with a 1080p window of text scrolling 8 px a frame, 120 frames at 60 fps, with a 16 Mbps cap. Quality was measured with ffmpeg's psnr filter against the source.

• Decoding: both streams decode with no errors in ffmpeg and carry identical color tags.
• Quality: x264's average is pulled down by the first ~30 frames after the keyframe (15–30 dB) and by dips mid-scroll (24.6 dB at frame 65). VideoToolbox holds 44–49 dB from its first frame.
• CPU: machine-wide CPU on the runner was noisy between runs (Spotlight and other background work), so the table uses the quietest run. voxl's own CPU was consistent.

The second frame of the scroll, 2x crop. Top: source. Middle: x264 (16 dB). Bottom: VideoToolbox (47 dB). By frame 60 x264 has caught up and the three look alike.

Not measured yet

• A real Mac. These numbers come from GitHub's virtual Mac, whose video encoder is paravirtualized. That's why x264 stays the default.
• Latency. VideoToolbox took ~16 ms per 2560x1600 frame there, against x264's ~5 ms. That keeps up with 60 fps but adds about 11 ms of latency. A real media engine should be much faster; if it isn't, the next step is encoding asynchronously (or with --stripes).
• CPU on a real Mac. Some of the whole machine's CPU above is probably the VM emulating the encoder.

Possible follow-ups once it's measured:

• Make VideoToolbox the default on macOS.
• Encode ScreenCaptureKit's NV12 surfaces directly, skipping the NV12→I420→NV12 copies (about 2 ms per frame of voxl's own CPU above).
• Raise --max-capture-pixels, which exists because x264's cost grows with area.

Testing

• cargo fmt --check, cargo clippy --all-targets --locked -- -D warnings and cargo test on Linux, macOS (CI) and Windows (CI); npm run check.
• New tests in src/mac/vt.rs, run by CI's macOS job:
  • encodes_annex_b_with_keyframes_on_request encodes a moving picture through VideoToolbox. It checks that every keyframe starts with SPS and PPS, that keyframes come exactly where they were asked for, and that delta frames carry slices and no parameter sets.
  • rewrites_avcc_as_annex_b covers the AVCC conversion, including a truncated length.
• One-off CI runs (since removed) decoded both encoders' streams with ffmpeg and produced the numbers and image above.

Want tembo to make any changes? Add a comment with @tembo and i'll get back to work!