Why TCP’s Handshakes, Headers, and Congestion Controls Really Matter

TCP vs UDP Basics

TCP is a connection‑oriented, reliable, byte‑stream transport protocol, while UDP is connection‑less and stateless. TCP’s three core advantages are connection establishment, reliability (stateful tracking and controllable retransmission), and byte‑stream handling.

Three‑Way Handshake

The handshake is illustrated with a romantic analogy, but the real process is:

Client sends SYN (entering SYN‑SENT).

Server replies SYN+ACK (entering SYN‑RECEIVED).

Client sends ACK (both sides reach ESTABLISHED).

This exchange confirms each side’s ability to send and receive data.

Four‑Way Termination

When a client wants to close a connection:

Client sends FIN → FIN‑WAIT‑1 (half‑close).

Server acknowledges with ACK → CLOSED‑WAIT.

Server sends its own FIN → LAST‑ACK.

Client acknowledges FIN → TIME‑WAIT for 2 MSL before fully closing.

Waiting 2 MSL ensures delayed packets don’t corrupt a new connection.

Half‑Open Queue and SYN Flood

Before the three‑way handshake, the server moves from CLOSED to LISTEN and creates two queues:

SYN (half‑open) queue holds connections in SYN‑RCVD state.

ACCEPT (full‑open) queue holds fully established connections ready for the application.

SYN Flood attacks flood the half‑open queue with forged SYN packets, exhausting resources and preventing legitimate connections. Mitigations include enlarging the SYN queue, reducing SYN‑ACK retries, and using SYN cookies.

TCP Header Fields

The header (bytes) includes source/destination ports, 32‑bit sequence number, acknowledgment number, flags (SYN, ACK, FIN, RST, PSH), window size (with scaling), checksum, and optional fields such as Timestamp, MSS, SACK, and Window Scale.

TCP Fast Open (TFO)

TFO reduces latency by allowing data to be sent in the initial SYN exchange. The client caches a SYN cookie from the server; on subsequent connections it sends SYN + cached cookie + data. The server can respond with data before the final ACK, saving one RTT.

TCP Timestamp Option

The 10‑byte option carries a timestamp (sender’s clock) and timestamp echo (receiver’s copy of the original timestamp). It solves two problems:

Accurate RTT measurement by using the original send time.

Distinguishing packets after sequence‑number wrap‑around.

Retransmission Timeout (RTO) Calculation

Two methods are described:

Classic Method

Smoothed RTT (SRTT) is updated with a smoothing factor α (≈0.8‑0.9). RTO = clamp(β × SRTT, lower, upper) where β≈1.3‑2.0.

Standard (Jacobson/Karels) Method

Updates SRTT with α=1/8, computes RTT variance (RTTVAR) with β=0.25, then RTO = μ × SRTT + δ × RTTVAR (μ≈1, δ≈4). This reacts faster to RTT changes.

Flow Control

Flow control uses the receiver’s advertised window (rwnd) and the sender’s congestion window (cwnd). The effective send window is min(rwnd, cwnd). The sender adjusts its window based on how much data the receiver has buffered.

Congestion Control

TCP maintains cwnd and ssthresh. Algorithms:

Slow Start : cwnd doubles each RTT until it reaches ssthresh.

Congestion Avoidance : after ssthresh, cwnd grows by 1 MSS per RTT (increase by 1/cwnd per ACK).

Fast Retransmit : on three duplicate ACKs, retransmit the missing segment immediately.

Selective Acknowledgment (SACK) : receiver tells sender which blocks were received, so only missing data is retransmitted.

Fast Recovery : after fast retransmit, set ssthresh = cwnd/2, cwnd = ssthresh, then increase cwnd linearly.

Nagle Algorithm and Delayed ACK

Nagle coalesces small outgoing segments: after the first segment, further data is sent only when either the MSS is reached or all previous data has been ACKed. Delayed ACK holds ACKs for up to 200 ms (max 500 ms) to combine them, reducing packet overhead. Certain conditions (large frames, quick‑ack mode, out‑of‑order packets) force immediate ACKs.

TCP Keep‑Alive

Keep‑alive probes detect dead connections. Linux defaults: probe after 7200 s, up to 9 probes, each 75 s apart. Many applications leave keep‑alive disabled because the interval is long and the feature adds overhead.