Unraveling TCP: From Handshakes to Congestion Control and Security

Can you explain the differences between TCP and UDP?

TCP is a connection‑oriented, reliable, byte‑stream transport protocol, while UDP is connection‑less and stateless.

TCP’s three core characteristics are:

Connection‑oriented – a three‑handshake establishes a connection before data exchange.

Reliability – the protocol tracks sent and received data (stateful) and can control retransmission.

Byte‑stream – data is treated as a continuous stream of bytes.

In contrast, UDP is stateless and uncontrollable.

Explain the TCP three‑way handshake and why it uses three steps

Love‑analogy illustration

The handshake is likened to a romantic exchange where each party confirms love and being loved before proceeding.

Real handshake process

Both ends start in CLOSED state. The server begins listening ( LISTEN). The client sends SYN and moves to SYN‑SENT. The server replies with SYN and ACK, entering SYN‑RECEIVED. The client acknowledges with ACK, reaching ESTABLISHED on both sides.

Any segment that requires acknowledgment consumes a TCP sequence number.

Why not two or four steps?

Two steps cannot confirm the receiver’s ability to accept data, while four steps add unnecessary delay and resource waste.

Explain the TCP four‑way termination

When the client wants to close, it sends FIN (state FIN‑WAIT‑1), becoming half‑closed. The server acknowledges ( CLOSED‑WAIT). After the client receives the ACK, it moves to FIN‑WAIT‑2. The server then sends its own FIN ( LAST‑ACK), and the client replies with ACK and enters TIME‑WAIT, waiting for 2 MSL before fully closing.

Purpose of waiting 2 MSL

Waiting ensures that delayed packets from the old connection do not interfere with a new connection that might reuse the same port.

Half‑open queue and SYN‑Flood relationship

Half‑open queue

When a client sends SYN, the server replies with SYN‑ACK and moves to SYN‑RCVD, placing the connection in the SYN (half‑open) queue.

Full‑open queue

After the client acknowledges, the connection moves to the accept queue, awaiting application acceptance.

SYN‑Flood attack

Attackers spoof many IPs and flood the server with SYN packets, filling the half‑open queue and exhausting resources.

Mitigation

Increase the SYN queue size.

Reduce SYN‑ACK retry count.

Use SYN cookies to avoid allocating resources before the final ACK.

TCP packet header fields

The header includes source and destination ports, sequence number, acknowledgment number, flags (SYN, ACK, FIN, RST, PSH), window size, checksum, and optional fields such as timestamps, MSS, SACK, and window scaling.

Sequence number and ISN

The 32‑bit sequence number tracks byte positions; the Initial Sequence Number (ISN) changes every 4 ms to prevent prediction attacks.

Flags

SYN and ACK have been covered; FIN ends a connection, RST forces an immediate reset, and PSH tells the receiver to deliver data to the application immediately.

TCP Fast Open (TFO) principle

TFO reduces latency by allowing the client to send data together with the initial SYN after a validated SYN cookie, enabling the server to respond with data before the three‑way handshake completes.

Role of TCP timestamps

Timestamps (10‑byte option) store the sender’s clock ( tsval) and echo the peer’s clock ( tsecr), enabling accurate RTT measurement and distinguishing packets when sequence numbers wrap.

How TCP timeout retransmission is calculated

Classic method

Smoothed RTT (SRTT) is updated as SRTT = α·SRTT + (1‑α)·RTT (α≈0.8‑0.9). RTO = min(ubound, max(lbound, β·SRTT)) with β≈1.3‑2.0.

Standard (Jacobson/Karels) method

Update SRTT: SRTT = (1‑α)·SRTT + α·RTT (α≈1/8). Compute RTT variance: RTTVAR = (1‑β)·RTTVAR + β·|RTT‑SRTT| (β≈0.25). Final RTO: RTO = μ·SRTT + δ·RTTVAR (μ≈1, δ≈4).

TCP flow control

Flow control uses a sliding window advertised by the receiver. The sender’s effective window is min(rwnd, cwnd). The receiver reduces its window when its buffer fills, causing the sender to throttle transmission.

TCP congestion control

Two key variables: congestion window ( cwnd) and slow‑start threshold ( ssthresh).

Slow start

cwnd starts small; each ACK increases cwnd by one MSS, doubling each RTT until cwnd ≥ ssthresh.

Congestion avoidance

After reaching ssthresh, cwnd grows linearly (increase by 1/cwnd per ACK), adding roughly one MSS per RTT.

Fast retransmit and fast recovery

Three duplicate ACKs trigger immediate retransmission (fast retransmit). The sender then halves ssthresh, sets cwnd = ssthresh, and grows cwnd linearly (fast recovery).

Nagle algorithm and delayed ACK

Nagle buffers small segments until either a full MSS is ready or all previous data is acknowledged, reducing packet overhead. Delayed ACK waits up to 200 ms to combine acknowledgments, but must send immediately for large frames, quick‑ack mode, or out‑of‑order packets.

Understanding TCP keep‑alive

Keep‑alive probes (default every 7200 s, 9 retries, 75 s interval) detect dead connections by sending empty packets; most applications keep it disabled because the interval is too long for typical use cases.