When the Third TCP Handshake Fails: Timeout, SYN Flood & Backend Lessons

TCP Three‑Way Handshake Overview

Both client and server start in the CLOSED state. The server begins listening, moving to LISTEN. The handshake proceeds as follows:

First handshake: client sends SYN (SYN=1, seq=x) and enters SYN_SEND.

Second handshake: server replies with SYN‑ACK (SYN=1, ACK=1, seq=y, ACKnum=x+1) and enters SYN_RECEIVED.

Third handshake: client sends ACK (ACK=1, ACKnum=y+1), entering ESTABLISHED; the server also moves to ESTABLISHED, allowing data transfer.

Consequences of a Missing Third Handshake

If the server does not receive the client’s final ACK, the connection remains in an incomplete state and cannot be used for data transmission. The server stays in SYN_RECEIVED, waiting for the ACK.

During this waiting period the server relies on the timeout retransmission mechanism. If the ACK never arrives, the server will repeatedly resend the SYN‑ACK until a retry limit is reached, after which the connection is considered failed and the server returns to CLOSED, releasing resources.

Timeout Retransmission and RTO Calculation

TCP uses a retransmission timeout (RTO) based on measured round‑trip time (RTT). The RTO is typically set slightly larger than the RTT to balance premature retransmissions against long delays.

RTO is computed using the Jacobson/Karels algorithm:

SRTT = (1 - α) * SRTT + α * RTT

RTTVAR = (1 - β) * RTTVAR + β * |RTT - SRTT|

RTO = μ * SRTT + δ * RTTVAR  =  SRTT + 4 * RTTVAR

Typical parameters are α = 0.125, β = 0.25, μ = 1, δ = 4.

SYN Flood Attack and Half‑Open Queue

When the final ACK is missing, it may be due to packet loss or a malicious SYN Flood attack. In a SYN Flood, attackers forge many source IPs and send massive SYN packets. The server replies with SYN‑ACK, but the forged addresses never return ACKs, filling the half‑open (SYN) queue and preventing legitimate connections.

Backend Design Lessons from TCP Handshake

The handshake embodies two useful ideas for backend systems:

State Confirmation : Both sides verify readiness before proceeding, similar to health checks or mutual authentication in microservice communication.

Timeout and Retry : Servers retry pending operations with a bounded limit, mirroring retry policies for HTTP calls, message queues, or database connections.

Applying these patterns improves reliability and resilience in distributed architectures.

Illustrations