Start with #16081 and #16495 together. Both address the burst-loss stall in #16393. Should a partial ACK start a second recovery episode before all data sent before the timeout has been acknowledged? Compare #15432 next: it changes how many packets the sender counts as still in flight.
Why start here? Two independent reproducers; three fixes that can make a stalled transfer finish.
| Reproducer | What it tells us |
|---|---|
| M003: Reno, SACK, no TCP timestamps | Stock stalled at byte 134,684 of 524,288 in five of five runs. With #16495 all five completed in 0.653–0.785 s. |
| Two-stack reproducer: CUBIC, SACK, RACK, host segmentation offload (GSO) | With no timestamps, three of eight stock fetches failed to finish in 120 s after losing the whole tail. With timestamps the reporter still saw a crawl. Each of the three fixes prevented the hang in this reproducer: #16081 and #15432 were tested on Tailscale’s pin, #16495 on master. |
On the same master revision, applying each PR’s relevant recovery changes separately shortened recovery with timestamps on. The reporter’s long-lived workload runs two warm-ups and then five lossy transfers over one connection per build:
| Sender | Time per measured transfer | Retransmission timeouts per transfer |
|---|---|---|
| Stock | 4.737–5.964 s | 21–27 |
| #16081 alone | 0.530–0.544 s | 1 |
| #15432 alone | 0.602–0.614 s | 1 |
| #16495 alone | 0.785–0.815 s | 2 |
Finishing this transfer doesn’t distinguish the three fixes. The partial-ACK and outstanding-packet cases below test where their behavior differs.
Even with GSO off, one of eight fetches hung on Tailscale’s pin. Neither replay establishes the cause on the original FreeBSD subnet router; its Tailscale version and the timestamp option in the SYN would help distinguish the cases.
Suggested sender review order
| Question | Read together | Decisive case |
|---|---|---|
| 1. When does timeout recovery end? | #16081 protects any timed-out flight; #16495 protects one whose timeout interrupted SACK recovery. | Try the reported partial ACK, then a timeout outside SACK recovery. #16419 covers a different transition: a second timeout without an advancing ACK. |
| 2. How many packets are still in flight? | #15634 preserves SACK credit across splits; #15432 subtracts packets marked lost by RACK. #14101 covers a SACK followed by a cumulative ACK outside recovery (#14092). | Test both directions. One repair counted twice: 4 sent with room for 5. Removing the extra count filled the window with GSO on or off. One probe copy missed: 8 sent with room for 7. |
| 3. Which timer is due? | #15785 puts a floor under probe delays at very short RTT; #15972 caps the probe deadline at the retransmission deadline even when that timer is not armed. | #16624 gives a successful repair of the oldest SACK hole a full retransmission timeout (RTO). Repairing a younger hole, or writing data that hasn’t gone on the wire, must not postpone it. |
| 4. What did the ACK actually prove? | #16659: a full SACK versus half a retransmission. #16660: new data as a probe beyond a full congestion window. | Use #15634’s partial-credit case on #16659. On #16660, check blocked writes and an ACK beyond sent data (also #16378). On #16596, an ACK of a retransmitted probe could acknowledge either copy. |
| 5. When should the congestion window grow or reset? | #16661: sparse application writes after loss. #16664: reselecting or switching controllers on a live socket. | Compare Reno and CUBIC on the same loss and ACK sequence, with both a backlogged sender and an application-limited one. |
Question 1 on the wire: which ACK distinguishes the two patches?
These Reno/SACK connections had timestamps disabled. A timeout interrupted SACK recovery; the sender transmitted new data, then timed out again. What mattered was how much of the old flight the receiver cumulatively acknowledged.
| ACK after the second timeout | What went on the wire |
|---|---|
| Partial ACK; SACK covers either just the last old byte or that byte and the newer segment | Linux, stock gVisor, #16081 and #16495 all repaired the one missing old byte and stayed in Loss with a two-packet congestion window. Extending the SACK across the old flight boundary did not distinguish the patches. |
| ACK of exactly the original flight; no SACK | Linux, stock gVisor and #16081 stayed in Loss; #16495 reported Open. All four retransmitted the newer segment. |
The arrangement of the final three old bytes matters. In three separate unacknowledged packets, they filled gVisor’s three-packet window; it sent the queued write within 2.3 ms of their ACK, without another application write. Linux had room for a fourth packet. SACKing the third packet before the write gave all four senders room for new data and made the partial-ACK comparison possible. Putting those bytes in one packet gave all four room for the exact-boundary comparison. These cases distinguish behavior during recovery; they do not measure throughput or a first timeout from Open.
This is a reading order, not a required merge order. The standard constructor selects Reno; there is also a CUBIC constructor. Both enable SACK and RACK by default; peers negotiate timestamp echoes separately. In six paired single-flow trials, corrected CUBIC delivered 1.62–2.36× Reno’s throughput at 100 Mbit/s, 100 ms RTT and 0.01% configured loss. Before changing the default, compare competing flows, shorter paths and queues, and no-loss controls.
Reviews that can proceed independently
| Symptom | Start with | What to check |
|---|---|---|
| Small segments stall behind a slow reader | #15144 · #15325 · #15433 | #15325 reports 480 → 4 drops at 536 bytes. But 257 repeated packets charged 165,492 bytes against a 4,096-byte budget. Compare ACKing rejects with #15433’s coalescing approach. |
| A corked close costs 200 ms | #16437 · #16671 | With 14 bytes buffered, stock gVisor waited out the timer on both close and SHUT_WR; Linux delivered the data and EOF promptly. |
| A zero-window probe cuts the congestion window | #16595 | If all sent data was acknowledged, the probe is for an unsent byte. Keep outstanding unacknowledged data as a separate control. |
| Offloaded traffic fails | #16666 · #14011 / #14579 | #16666 concerns the first send after restore on a new route. #14011 concerns forwarding an aggregate larger than the outgoing MTU whose individual segments fit. These need different tests. |
| A closed connection keeps its port | #16102 / #16180 | Bind a dual-stack wildcard, connect to a mapped IPv4 peer, then close with a reset. On #16180, IPv4 can rebind but IPv6-only gets EADDRINUSE. Linux releases both. |
More specialized reviews and work already on master
- Minimum RTT (#16381): Changing the handshake seed did not change the probe decision in any of the 40 fetches on the reporter’s baseline: 35 recovered before timeout; five timed out once. All three variants used the same estimator for data ACKs.
- Which copy arrived? In eight matched pairs, the receiver sent byte-identical ACKs whether the original packet or its probe reached it, even with timestamps. At 20 ms after the probe, a recent 10 ms minimum allowed both preceding losses to be repaired; with the minimum refreshed to 30 ms, neither repair was sent in the next 10 ms. Both losses were real. Relaxing RACK’s retransmission check would also accept cases where the probe never arrived.
- Reordering: #16602 and #16605 govern when the RACK reordering allowance decays or saturates. #16600 covers timestamp wrap; #15651 covers stale timestamps on input. Neither timestamp case explains a trace without timestamps.
- Spurious recovery: #16601 checks that timestamps were negotiated and the ACK carries the option before attempting Eifel detection; #16258 keeps duplicate-SACK history per connection. #16403 generates D-SACK for a duplicate below the cumulative ACK. #14458 addresses the window after spurious recovery is detected. Test detection and undo separately.
- Closing and reopening sockets: #16377 makes the unread FIN explicit in the window-edge tests; the follow-up checks the ACK when the FIN is still queued. #15013 / #15019 is a different path: a new SYN reuses a connection in TIME_WAIT through
tcp.Forwarder. - Software segmentation: #15634 covers splitting a lost write. Two further measured cases are choosing the highest outstanding segment for a tail probe and advancing only over bytes the link accepted. These experiments are on the fork. For forwarded traffic, also see the container report; forwarding needs a capable outgoing route, not just copied metadata (#13386).
- Already on master:
#15575 retransmission cursor·#15633 SACK loss threshold·#15618 RACK initialization·#15819 ambiguous RTT samples·#15926 Nagle and full segments·#16055 RACK reordering test·#15577 CUBIC recovery tests·#15579 CUBIC ACK credit·#15604 CUBIC epoch initialization·#16336 SACK on SYN cookies·#16382 advertised receive window.
This index is updated in place. Completed entries are struck through; replacements link to the PR that superseded them. Please put new traces on the relevant bug and code-level feedback on the PR.
Maintained by Codex on behalf of @tamird.
Activity
0No activity yet.