Why Knowing TCP’s Three‑Way Handshake and Four‑Way Teardown Isn’t Enough

Introduction

TCP’s three‑way handshake and four‑way termination are basic interview questions, but real‑world troubleshooting requires a deeper understanding of each state, flag, and kernel parameter that influences TCP behavior.

TCP Header Overview

Key fields in the TCP header include:

Sequence Number : the first byte’s sequence, used for ordering.

Acknowledgment Number : the next expected byte, confirming receipt.

Flags : SYN, ACK, FIN, RST, PSH, URG.

Window : receiver’s buffer size for flow control.

TCP State Diagram

The main states are CLOSED, LISTEN, SYN_SENT, SYN_RCVD, ESTABLISHED, FIN_WAIT_1, FIN_WAIT_2, CLOSE_WAIT, CLOSING, LAST_ACK, TIME_WAIT, and CLOSE_WAIT. Each state’s meaning and typical scenario are listed in the original table.

Three‑Way Handshake Details

Step 1 – SYN

Client sends SYN=1, Seq=x (initial sequence number is random).

Client enters SYN_SENT state.

Step 2 – SYN‑ACK

Server replies SYN=1, ACK=1, Seq=y with Ack=x+1.

Server enters SYN_RCVD state.

Step 3 – ACK

Client sends ACK=1, Seq=x+1, Ack=y+1.

Both sides move to ESTABLISHED.

Why Three Steps?

TCP is full‑duplex; both directions must be acknowledged. A two‑step handshake would allow a delayed SYN to be mistaken for a new connection, wasting resources. The third ACK ensures the server only establishes the connection after the client’s final acknowledgment.

Four‑Way Handshake?

Four steps are unnecessary because the SYN and ACK can be combined, making three the minimal reliable exchange.

ISN Randomness

Random Initial Sequence Numbers prevent TCP sequence‑prediction attacks, making it hard for an attacker to forge a valid connection.

Inspecting ISN and Window Scaling

# View current TCP sequence numbers
cat /proc/net/tcp
cat /proc/net/tcp6
# Use ss for detailed info
ss -ti state established
# Use netstat
netstat -tn | grep ESTABLISHED

# View window scaling factor
cat /proc/sys/net/ipv4/tcp_window_scaling
# Adjust temporarily
sysctl -w net.ipv4.tcp_window_scaling=1

Four‑Way Termination Details

Step 1 – FIN

Active closer sends FIN=1, Seq=u and enters FIN_WAIT_1.

Step 2 – ACK

Passive side acknowledges, entering CLOSE_WAIT.

Step 3 – FIN

Passive side sends its own FIN, moving to LAST_ACK.

Step 4 – ACK (TIME_WAIT)

Active side acknowledges and stays in TIME_WAIT for 2 MSL (typically 60 s) to ensure the final ACK is received and old packets disappear.

TIME_WAIT Analysis

Ensures the final ACK reaches the passive side.

Allows old packets to expire before a new connection reuses the same tuple.

On Linux the timeout is controlled by net.ipv4.tcp_fin_timeout (default 60 s). The state usually lasts 120 s.

Kernel Parameters for TCP

# View TIME_WAIT timeout
cat /proc/sys/net/ipv4/tcp_fin_timeout
# Enable reuse of TIME_WAIT sockets (requires timestamps)
cat /proc/sys/net/ipv4/tcp_tw_reuse
sysctl -w net.ipv4.tcp_tw_reuse=1
# Adjust FIN timeout
sysctl -w net.ipv4.tcp_fin_timeout=30
# Increase TIME_WAIT bucket limit
sysctl -w net.ipv4.tcp_max_tw_buckets=100000

Handling Excessive TIME_WAIT

Enable tcp_tw_reuse (requires tcp_timestamps=1).

Reduce tcp_fin_timeout.

Increase tcp_max_tw_buckets.

Prefer keep‑alive or HTTP keep‑alive for short‑lived connections.

CLOSE_WAIT Accumulation

Usually caused by applications not calling close() or connection‑pool misconfiguration.

# List CLOSE_WAIT sockets
ss -tup state close-wait
# Find owning processes
ss -tup | grep CLOSE_WAIT
# Inspect Java, Python, Go processes for missing close calls

RST Packets

RST indicates an abnormal termination. Common causes:

Target port not listening.

SO_LINGER set to 0 (forces RST on close).

Invalid sequence numbers.

Application crash.

# Capture RST packets
tcpdump -i eth0 'tcp[tcpflags] == tcp-rst'
# Count RSTs
netstat -s | grep -i reset

tcpdump Usage

# Capture all TCP traffic
tcpdump -i eth0 tcp
# Capture specific port
tcpdump -i eth0 port 80
# Capture SYN packets only
tcpdump -i eth0 'tcp[13] & 2 != 0'
# Capture handshake packets
tcpdump -i eth0 -nn -tttt 'port 8080' | grep -E '(SYN|ACK|FIN)'
# Example three‑way handshake output
10:15:23.456789 IP 192.168.1.10.45678 > 192.168.1.20.8080: Flags [S], seq 1000, win 65535, options [mss 1460], length 0
10:15:23.456792 IP 192.168.1.20.8080 > 192.168.1.10.45678: Flags [S.], seq 2000, ack 1001, win 65535, options [mss 1460], length 0
10:15:23.457001 IP 192.168.1.10.45678 > 192.168.1.20.8080: Flags [.], ack 2001, win 65535, length 0

Connection Queues

SYN (Half‑Open) Queue

When the server receives a SYN it creates an entry in the SYN queue (size controlled by net.ipv4.tcp_max_syn_backlog, default 128). Use ss -ltn state syn-recv to view.

Accept (Full) Queue

After the handshake the connection moves to the accept queue (backlog size net.core.somaxconn, default 128). Use ss -ltn to see Recv‑Q and Send‑Q.

Queue Tuning Recommendations

Set tcp_max_syn_backlog ≥ 2048.

Set somaxconn ≥ 2048.

Configure application listen backlog ≥ 1024 (e.g., Nginx backlog=65535, Go net.Listen uses system backlog, Tomcat acceptCount, Python listen(2048)).

Common Production Issues & Troubleshooting

Connection Drops Immediately

Symptoms: client receives RST after connect.

Possible causes: port not listening, full accept queue, firewall, server crash.

# Check listening port
ss -tlnp | grep 8080
# Check firewall rules
iptables -L -n
firewall-cmd --list-all
# Inspect TCP states
ss -ti state established
# Capture packets
tcpdump -i eth0 port 8080 -nn

Excessive CLOSE_WAIT

Cause: application fails to close sockets.

# List CLOSE_WAIT sockets
ss -tup state close-wait
# Identify owning processes
ss -tup | grep CLOSE_WAIT
# Examine code paths, timeouts, keepalive settings

Too Many TIME_WAIT

# Count TIME_WAIT sockets
ss -tan state time-wait | wc -l
# Show source IP distribution
ss -tan state time-wait | awk '{print $4}' | cut -d: -f1 | sort | uniq -c | sort -rn
# Apply reuse and timeout tuning (see kernel parameters above)

Port Exhaustion

Short‑lived connections consume ephemeral ports.

# View port range
cat /proc/sys/net/ipv4/ip_local_port_range
# Expand range if needed
sysctl -w net.ipv4.ip_local_port_range="1024 65535"
# Enable TIME_WAIT reuse
sysctl -w net.ipv4.tcp_tw_reuse=1

Half‑Open Connections (SYN Flood)

# Monitor SYN‑RECV count
ss -ltn state syn-recv | wc -l
# Increase backlog to mitigate
sysctl -w net.ipv4.tcp_max_syn_backlog=2048

Monitoring Scripts

TCP Connection Statistics

#!/bin/bash
# tcp_status_monitor.sh

echo "=== TCP Connection Statistics ==="
ss -tan | awk '{print $1}' | sort | uniq -c | sort -rn

timewait_count=$(ss -tan state time-wait | wc -l)
echo "Current TIME_WAIT: $timewait_count"
if [ $timewait_count -gt 50000 ]; then
  echo "Warning: TIME_WAIT count high"
fi

closewait_count=$(ss -tan state close-wait | wc -l)
echo "Current CLOSE_WAIT: $closewait_count"
if [ $closewait_count -gt 1000 ]; then
  echo "Warning: CLOSE_WAIT count abnormal"
fi

synrecv_count=$(ss -tan state syn-recv | wc -l)
echo "Current SYN_RECVD: $synrecv_count"
if [ $synrecv_count -gt 1000 ]; then
  echo "Warning: SYN queue may be under attack"
fi

ss -tan state established | awk '{print $4}' | cut -d: -f1 | sort | uniq -c | sort -rn | head -10
cat /proc/net/netstat | grep TcpExt | tail -1

Port‑Specific Monitoring

#!/bin/bash
# port_conn_monitor.sh
PORT=${1:-80}
THRESHOLD=${2:-1000}

echo "=== Port $PORT Connection Statistics ==="
ss -tunlp | grep ":$PORT " | head -5

est_count=$(ss -tan "sport = :$PORT or dport = :$PORT" state established | wc -l)
echo "Current Established: $est_count"
if [ $est_count -gt $THRESHOLD ]; then
  echo "Warning: connections exceed threshold $THRESHOLD"
fi

ss -tan "sport = :$PORT or dport = :$PORT" | awk 'NR>1 {print $1}' | sort | uniq -c | sort -rn

Summary

Understanding TCP state transitions, the rationale behind the three‑way handshake, and the mechanics of the four‑way termination is essential for diagnosing real‑world network problems. By combining kernel parameter tuning, command‑line inspection tools ( ss, netstat, tcpdump), and simple monitoring scripts, operators can quickly locate root causes such as excessive TIME_WAIT, lingering CLOSE_WAIT, SYN floods, or port exhaustion.