TCP over long fat pipes

A link with high bandwidth and high latency is called a long fat network, and it defeats TCP’s defaults in a specific, calculable way.

The arithmetic

TCP will not send more unacknowledged data than the receive window. Throughput is therefore window divided by round-trip time.

A 10 Mbit/s link with 600 ms round trip needs a window of 10,000,000 × 0.6 / 8 = 750 KB to be filled. The classic TCP window field is 16 bits — a maximum of 65,535 bytes. Without window scaling you get about 850 kbit/s on a 10 Mbit/s link, and every diagnostic will show the link is nowhere near full.

Window scaling

RFC 1323 window scaling multiplies the window by a negotiated factor, allowing windows up to a gigabyte. It is negotiated in the SYN and cannot be added later.

The failure worth knowing about: some middleboxes strip or mangle the option. Both ends believe they agreed something the other did not, and throughput stays poor. Confirm it from a capture of the handshake rather than from the host’s configuration.

sysctl net.ipv4.tcp_window_scaling      # should be 1
sysctl net.ipv4.tcp_rmem net.ipv4.tcp_wmem
tcpdump -ni eth0 'tcp[tcpflags] & tcp-syn != 0' -v | grep -i wscale

Loss is punished twice

TCP treats loss as congestion and halves its rate. Recovery takes many round trips, and on a 600 ms path that is a long time. A satellite link with even 1% loss will perform far worse than the same loss on a terrestrial link.

This is why error correction on satellite links matters so much: preventing a lost packet is worth far more than recovering from one.

Congestion control

BBR estimates bandwidth and round-trip time rather than treating loss as congestion, and performs markedly better on long, lossy paths. Where you control both ends, it is usually the single most effective change available.

sysctl -w net.ipv4.tcp_congestion_control=bbr