#35

feat: stop one slow viewer from lowering quality for everyone

OpenDarrenBaldwin07wants to merge intofrom

What

A viewer that can't keep up no longer slows everyone else down. Until now, every viewer (and every /record) shared one encoder: the bitrate followed the slowest viewer (RateCtl::update took the minimum), and flow_ok held captures back for whoever was furthest behind. Someone watching from a phone or bad Wi-Fi lowered the picture, and on a slow enough link the frame rate, for everyone.

Now such a viewer moves to a second, lower-bitrate stream. That stream only exists while someone needs it. Capture and conversion stay shared; each stream has its own encoder, rate control, flow control, paint-over and refinement state. A stream whose viewers are catching up skips captures and collects what changed (rows and macroblocks) until it can send again.

--slow-viewer-stream=false turns it off. One viewer costs exactly what it did before.

Results

16 vCPU VM, 2560x1600 Xvfb, 1080p animation (bench/loadgen), bench/nclient.py viewers. The slow viewer goes through bench/shaper.py with 60 ms RTT. Cells are FPS (worst second), latency p50/p99 and bitrate. Old and new builds were run interleaved.

CaseFast viewer, beforeFast viewer, afterServer CPU (cores)
Fast viewer alone60.0 (59.6), 10/16 ms, 7.21 Mbps60.0 (59.8), 10/13 ms, 7.24 Mbps1.04 → 1.02
+ a 5 Mbps viewer59.7 (57.9), 11/18 ms, 4.55 Mbps60.0 (59.7), 11/18 ms, 7.17 Mbps1.15 → 2.16
+ a 1.5 Mbps viewer23.1 (14.8), 14/28 ms, 1.29 Mbps59.9 (58.9), 11/24 ms, 7.15 Mbps0.56 → 1.48
5 Mbps viewer first, fast one joins 8 s later59.8 (56.9), 11/15 ms, 4.48 Mbps60.0 (59.9), 11/17 ms, 7.11 Mbps1.15 → 2.09
  • The slow viewer gets the same throughput and frame rate as before (e.g. 4.51 → 4.59 Mbps at 5 Mbps, 22.9 → 22.6 FPS at 1.5 Mbps). Moving costs it one keyframe, a brief stall on its link: the worst second dropped to 41 FPS when the move happened mid-measurement.
  • Recordings: /record stays on the main stream. A 15 s recording made while only a 3 Mbps viewer watched was 2.80 Mbps before and 7.00 Mbps now, the same quality as with a fast viewer.
  • CPU: the second encode costs about one core while it runs. With one viewer, or with viewers that all keep up, nothing changes.
  • No false splits: two fast viewers, and a fast viewer with one on a 20 Mbps / 60 ms link, stayed on one stream.

How viewers move

Tiering in src/video.rs looks at the viewers every 100 ms:

  • Out: a viewer's congestion (the check RateCtl already uses) adds to a decaying "strain" with a half-life of about 14 s. It moves to the slow stream once its strain reaches 3 while another main-stream viewer has been watched for 1 s without congesting. Congestion has to be remembered: once the shared stream has slowed down to suit a viewer, that viewer rarely congests again, and a fast viewer joining later would otherwise wait for the next probe (10–18 s in testing, versus ~1 s now).
  • The new stream starts just under the slow viewer's measured throughput (RateCtl::seeded). The main stream forgets that viewer's capacity and ramps straight back up (RateCtl::reprobe).
  • Back: a viewer rejoins once it has gone 10 s without congestion and has received at least 80% of what the main stream's viewers get, measured while the main stream carries more than the bitrate floor. The 10 s doubles with every move (up to 16x), so a borderline link settles instead of flapping. If the main stream's viewers all leave, the slow viewers take it over with their own rate controller, and the second encoder stops.
  • Switching streams: Client::move_to and the encoders' sends both run under the viewer's inflight lock, so nothing from the old stream can follow the new stream's keyframe. Frame IDs come from one counter (Hub::frame_id), so a viewer never sees an ID twice. The web client needs no changes: it already starts over at any keyframe.

Smaller changes:

  • The two capture buffers are shared as Arc<I420>; a buffer is reused once every encoder that took it has handed it back.
  • Encoders open on their own thread, so starting the slow stream doesn't stall the capture loop.
  • /stats adds slow_target_kbps and each client's tier.
  • The README gains the flag and a paragraph.

Testing

  • cargo fmt --check, cargo clippy --all-targets --locked -- -D warnings and cargo test with CI's nixpkgs toolchain; npm run check.
  • New unit tests for Tiering:
    • the viewer holding the others back moves, after the other viewer has been watched for a while;
    • nobody moves when alone, when everyone is congested, or with the flag off;
    • rejoining needs both the calm period and keeping pace;
    • slow viewers take over the main stream when the others leave.
  • End to end, with fast and shaped viewers: the cases above, plus:
    • a 3 Mbps link that recovers after 20 s: the viewer moves back ~17 s later, once;
    • a link that never recovers: it stays put for 60 s;
    • the fast viewer leaving;
    • /record with a slow viewer;
    • --stripes=2;
    • a dynamic-resolution resize while a viewer is on the slow stream: both viewers get the new size and keep streaming;
    • --slow-viewer-stream=false, which behaves like before.
  • The real web client in headless Chromium, through a 3 Mbps link, with a fast viewer next to it: it decodes the slow stream at 58 FPS / 3 Mbps, then switches back to the main stream's encoder when the fast viewer leaves, with no decoder errors.

Not tested on macOS or Windows. The change is in the platform-neutral src/video.rs and src/hub.rs.


Want tembo to make any changes? Add a comment with @tembo and i'll get back to work!