Mastering TCP: How the Three‑Way Handshake and Socket States Work

TCP Three‑Way Handshake

TCP is a connection‑oriented protocol that requires a three‑packet exchange (the three‑way handshake) to establish a connection before data can be sent, and a four‑packet exchange to close it.

Connection Establishment

Both endpoints start in the CLOSED state. The client (A) initiates the connection, and the server (B) listens for requests.

First handshake: A sends a SYN packet (seq=x) and enters SYN‑SENT .

Second handshake: B replies with SYN‑ACK (seq=y, ack=x+1) and enters SYN‑RCVD .

Third handshake: A sends ACK (ack=y+1, seq=x+1), possibly carrying data, and both sides move to ESTABLISHED .

TCP Header

TCP State Summary

ESTABLISHED : Connection is active.

SYN_SENT : Client has sent SYN, awaiting reply.

SYN_RECV : Server has received SYN, awaiting ACK.

FIN_WAIT1 : Initiating side has sent FIN.

FIN_WAIT2 : Waiting for the remote FIN.

TIME_WAIT : Waiting for delayed packets to expire.

CLOSE_WAIT : Remote side closed, waiting to close locally.

LAST_ACK : Waiting for final ACK after sending FIN.

LISTEN : Server is listening for connections.

CLOSING : Both sides have sent FIN but not yet acknowledged.

UNKNOWN : Undefined state.

SYN Flood Attack

During the handshake, a server that has sent SYN‑ACK but not yet received ACK is in a half‑open state (SYN_RECV). An attacker can flood the server with forged SYN packets, filling the half‑open queue, causing denial‑of‑service. Detection is simple: many SYN_RECV entries with random source IPs.

netstat -n | awk '/^tcp/ {++sam[$NF]} END {for(num in sam)print num,sam[num]}'
TIME_WAIT 30
FIN_WAIT1 1
ESTABLISHED 615
SYN_RECV 2

Linux TCP Kernel Parameter Optimization

Two ways to modify parameters:

Echo values directly to /proc/sys/... (temporary).

Add entries to /etc/sysctl.conf and apply with sysctl -p (persistent).

Relevant directories: /proc/sys/net/ipv4/, /proc/sys/net/ipv4/netfilter/ (requires firewall), and /proc/sys/net/core/.

Client State Transitions

CLOSED → SYN_SENT → ESTABLISHED → FIN_WAIT_1 → FIN_WAIT_2 → TIME_WAIT → CLOSED

Server State Transitions

CLOSED → LISTEN → SYN_RECV → ESTABLISHED → CLOSE_WAIT → LAST_ACK → CLOSED