Why TCP Congestion Control Adds Unexpected RTTs and How to Diagnose It

Problem Scenario

In a low‑latency environment (<1 ms network RTT, 2 ms service latency) a migration to a new network with 100 ms RTT caused observed service latency to rise to 300 ms–400 ms, far higher than the expected ~102 ms.

TCP Handshake Overhead

The three‑way handshake itself adds one RTT. The packet exchange is:

+0   A → B SYN
+0.5RTT B → A SYN+ACK
+1   A → B ACK
+1   A → B Data

Since the client can start sending data after the ACK, only one additional RTT is incurred for connection establishment.

Slow Start and Initial Congestion Window

After the connection is established, TCP does not send all data at once. It sends a limited amount determined by the initial congestion window (cwnd) to avoid overwhelming the network. Linux defaults to TCP_INIT_CWND = 10, meaning up to 10 × MSS (≈14 KB) can be sent in the first RTT. #define TCP_INIT_CWND 10 If the payload is ≤ 14 480 bytes, the transfer completes in two RTTs (one for the handshake, one for data). Any byte beyond this limit forces an extra RTT.

Experimental Verification

Packet captures in a 100 ms RTT environment show:

Sending exactly 14 480 bytes takes 200 ms (2 RTT).

Sending 14 481 bytes takes 300 ms (3 RTT).

Impact on Service Latency

For a typical request/response of ~30 KB each, both directions exceed the initial cwnd, resulting in four RTTs: one for the handshake, one for request data, one for response data, and an extra RTT for the second slow‑start burst. In a 100 ms RTT network this explains observed latencies around 400 ms.

Mitigation

Using persistent TCP connections (keep‑alive) eliminates the handshake overhead for subsequent requests, reducing the total RTT count.

Discussion on Tuning initcwnd

Some developers attempt to set the initial cwnd via socket options:

setsockopt(fd, IPPROTO_TCP, TCP_CWND, &val, sizeof(val));

However, this approach is controversial. IETF experts argue that initcwnd reflects link characteristics and should be configured by network operators, not applications, because an overly large value can exacerbate congestion and prevent automatic recovery.

Linux allows changing the default via route configuration, e.g., using ip route to set a larger cwnd, which can be verified by observing fewer RTTs in packet captures.