Mastering TCP: Flow Control, Slow Start, and Window Tuning for Linux

Many people feel confused about TCP optimization, but understanding TCP’s operation demystifies it. Ilya Grigorik’s “High Performance Browser Networking” provides detailed explanations, which this article summarizes in a more accessible way.

Flow Control

When the sender transmits more data than the receiver can handle, packet loss occurs. Flow control requires both sides to announce their receive window (rwnd), indicating how much data the receiver can buffer. If the window shrinks to zero, the receiver is full and must process data before more arrives.

From the user’s perspective, when browsing a webpage the client is the receiver and the server the sender; when uploading, the roles reverse.

Slow Start

Flow control prevents receiver overload but not network overload. Slow start introduces the congestion window (cwnd), an internal sender parameter that grows exponentially as acknowledgments are received, probing the network capacity.

During slow start, cwnd growth can cause network congestion, manifested as packet loss; the window then rapidly reduces to allow recovery.

In practice, the amount of unacknowledged data is limited by the smaller of rwnd and cwnd.

Congestion Avoidance

When cwnd exceeds a threshold (ssthresh), growth changes from exponential to linear to avoid congestion.

Adjusting rwnd to a Reasonable Value

Slow transfer speeds often stem from an improperly sized rwnd. The optimal rwnd equals the Bandwidth‑Delay Product (BDP). For a 10 Mbps link with 100 ms latency: BDP = 10 Mbps × 100 ms = (10/8) × (0.1) = 0.125 MB TCP’s 16‑bit window field caps at 64 KB; larger windows require TCP window scaling.

Linux controls rwnd via kernel parameters:

sysctl -a | grep mem
net.ipv4.tcp_rmem = <MIN> <DEFAULT> <MAX>

The kernel can auto‑tune the buffer between minimum and maximum values. The auto‑tuning status is checked with: sysctl -a | grep tcp_moderate_rcvbuf If auto‑tuning is disabled, set the default to BDP; if enabled, set the maximum to BDP.

Buffer overhead must also be considered:

Buffer / 2^tcp_adv_win_scale

Depending on tcp_adv_win_scale (1 or 2), the effective buffer size is:

BDP / (1 – 1 / 2^tcp_adv_win_scale)

RTT (used for BDP) can be measured with ping or, if ICMP is blocked, with a SYN‑ACK probe.

Adjusting cwnd to a Reasonable Value

The initial cwnd is typically:

min(4 × MSS, max(2 × MSS, 4380))

With a standard Ethernet MSS of 1460 bytes, the initial cwnd is 3 MSS.

For small transfers (e.g., a 20 KB webpage), a small initial cwnd can limit performance, as shown below.

Increasing the initial cwnd dramatically reduces the number of RTTs needed:

Google’s research suggests a default of 10 MSS. On modern Linux you can set it with:

ip route | while read p; do
    ip route change $p initcwnd 10;
 done

Note that cwnd cannot exceed rwnd; a small rwnd will limit cwnd’s effect.

To verify a server’s initcwnd, capture packets during the handshake. In the example, RTT is 168 ms, the first data packet arrives at 409 ms, giving an initial cwnd of 2 MSS.

Be aware that NIC offloading may affect packet counts; disabling offloading may be necessary.