Very slow Reno recovery after packet loss without TCP timestamps

@mapelkon, could you share the standalone reproducer and the seed or packet-drop sequence from one stalled run? The two guards do different things: the ACK path still marks packets lost, while the reorder timer skips loss detection entirely during RTO recovery. Trying each guard separately on the same burst would tell us which one affects the stall.

@ayushr2, there is evidence for trying CUBIC as the default. In six paired single-flow trials on a 100 Mbit/s link with 100 ms RTT and 0.01% configured loss, our corrected CUBIC branch delivered 1.62–2.36 times Reno’s throughput. Both algorithms use the recovery paths in this report, so I’d keep the Reno bug open regardless of the default.

Posted by Codex for @tamird.

@ayushr2 nothing I can do here. I have a bunch of open fixes - getting all of them reviewed has been slow. I'll let the bots collect what relevant information is available here.

Thanks. Attached is the standalone synthetic reproducer and the actual drop sequence from stalled trial M003.

Context: We investigated severe browser loading stalls through a Tailscale subnet router on FreeBSD. Users sometimes needed up to 15 reloads to complete an enterprise application's login flow. Client and gateway captures showed prolonged TCP recovery despite an open client receive window. We then reproduced a similar recovery pattern using synthetic, memory-only gVisor tests. This does not establish the original packet-loss cause; no company captures are included.

Synthetic setup: gVisor master 7bffefa9185aa55ba377e8a5534210a7f0cbf266 (generated Go snapshot 0f7a9693a2d91e4682d6ae0a133bba87ac583aca), Go 1.27.3, Windows/amd64; Reno, SACK, no TCP timestamps, 512 KiB transfer. Same pinned revision, compiler and harness; five sequential burst trials per variant, alternating order. All four no-loss controls passed.

Every completed transfer passed payload SHA256. One stock completion still took 26.1 seconds. In this small sample, ACK-only avoided the long stall; timer-only did not. This is diagnostic evidence, not validation of a production fix.

There is no random loss seed: drop first transmissions of distinct data-segment ordinals 101–124. M003 reached only 101–122; drop-sequence.json lists their relative sequence offsets and lengths. The fixed rule is shared across variants, but scheduling/packetization vary; this is not an exact ACK/timing replay.

AI assistance: Codex assisted the analysis, test/patch code and this reply. No independent human TCP review has been performed.

gvisor-16393-reproducer-final.zip

@mapelkon, thanks for sending M003. We ran your program unchanged on Linux/AMD64 against the gVisor revision you tested. All five stock Reno transfers stalled at 134,684 of 524,288 bytes after seven timeouts. They reached dropped segments 101–122, just as M003 did.

#16495 fixes what happens when a retransmission times out during SACK recovery. A partial ACK could make the sender start a second recovery episode for the same flight and cut its slow-start threshold again. The fix keeps the original recovery point until the peer acknowledges that flight in full; it leaves timer-driven loss detection enabled.

With that fix, all five transfers finished in 0.653–0.785 seconds and delivered the correct payload. Your ACK-only patch also finished all five (0.651–0.878 seconds). All three no-loss controls passed. Full results, including the five stock failures.

This reproduces the synthetic stall. We haven't established whether the same sequence occurred on the FreeBSD router.

Posted by Codex for @tamird.

Tailscale's netstack shows a related crawl (tailscale/tailscale#21946; I see stalls like it through tailscale serve): after a burst loss, it repairs one or two segments about every 200 ms. In a two-stack reproduction, netstack-rack-repro, a client with TCP timestamps gets that crawl, and a client without them turns it into the increasingly long waits described here. (With timestamps the burst is also smaller, about 25–110 packets against about 1000, because the sender had already left slow start during the warm-up fetches; without timestamps it had not. The RTO difference below does not depend on that.) The sender is set up as Tailscale sets up netstack (CUBIC, SACK, RACK, host GSO), the path is 3 ms at 80 Mbit/s, and each measured 2.8 MB response, sent after 3 s of idle time, gets a scripted burst loss.

With gVisor master 4777ee78 and a client without timestamps, 3 of 8 fetches, those whose burst took the whole tail (about 1000 packets), had not finished after 120 s; the other five finished in about 0.3 s without an RTO. With the gVisor that Tailscale 1.104 pins, 4 of 8 hung. This is with CUBIC, which Tailscale sets, so in this scenario changing gVisor's default from Reno would not avoid it. In each hung fetch, the sender retransmitted about two segments per RTO, and the RTO doubled at every expiry: 102 s after 9 RTOs, with 20 segments retransmitted.

Without timestamps, gVisor applies Karn's rule (RFC 6298, section 3): a retransmission moves RTTMeasureSeqNum to SndNxt (every retransmission on master since #15819, each fast retransmit on the pin), and only an ACK beyond it gives an RTT sample. During the crawl nothing new is sent, so no ACK gives one, and the backed-off RTO is never recomputed: RFC 6298 collapses it only with a new RTT measurement, "which can only happen when new data has been sent and acknowledged" (section 5). That would explain the "increasingly long waits between small repairs" here. With timestamps, the RTO goes back to 200 ms after every expiry in the same scenario's crawl, because the timestamped ACK of each retransmission gives a sample.

In the reproduction, each of these ends the hang, and every fetch finished:

#16081 keeps the sender from starting a second recovery for the flight that an RTO is already repairing, and #16495 does the same when the RTO interrupts SACK recovery, as it does in these trials. #15432 stops RACK-lost segments from counting as in flight, so the repair is not limited to about two segments per RTO.

@mapelkon, your stalls went through a Tailscale subnet router on FreeBSD. tailscaled handles subnet routes in netstack there by default, without GSO. Since Tailscale 1.104 that netstack runs CUBIC with RACK on; 1.82 through 1.102 ran it with RACK off (Reno from 1.82.5), and 1.80 and earlier ran Reno with RACK on, as in your synthetic test. In the reproduction, with the pinned gVisor, no GSO and a client without timestamps, 1 of 8 fetches hung the same way. Which Tailscale version did the router run, and do the clients' SYNs in your captures carry the TCP timestamp option? Windows does not offer it by default.