Beyond Interview Answers: Mastering TCP’s Three‑Way Handshake and Four‑Way Teardown

Introduction

Understanding TCP state transitions is essential for real‑world troubleshooting; memorising interview answers is insufficient when production issues arise.

1. TCP Protocol Basics

1.1 Header Fields

Sequence Number : the first byte number of the segment, used for ordering.

Acknowledgment Number : the next expected byte, confirming receipt.

Flags : SYN, ACK, FIN, RST, PSH, URG.

Window : receiver’s advertised buffer size for flow control.

1.2 State Overview

CLOSED : initial or fully closed state.

LISTEN : server waiting for connections.

SYN_SENT : client has sent SYN.

SYN_RCVD : server has received SYN and sent SYN‑ACK.

ESTABLISHED : normal data transfer.

FIN_WAIT_1 : active closer sent FIN, awaiting ACK.

FIN_WAIT_2 : active closer received ACK, awaiting peer FIN.

CLOSE_WAIT : passive closer received FIN, waiting for application close.

LAST_ACK : passive closer sent FIN, awaiting final ACK.

TIME_WAIT : ensures the final ACK is received and old packets expire.

CLOSING : both sides have sent FIN simultaneously.

2. Three‑Way Handshake

2.1 Handshake Process

Client                              Server
    |                               |
    |--- SYN=1, Seq=x -------------> |   (first handshake)
    |                               |
    |<--- SYN=1, ACK=1, Seq=y ------ |   (second handshake, Ack=x+1)
    |                               |
    |--- ACK=1, Seq=x+1 ----------> |   (third handshake, Ack=y+1)
    |                               |
    |========= ESTABLISHED =========|

First handshake : client sends SYN with an initial sequence number (ISN).

Second handshake : server replies with SYN+ACK, providing its ISN and acknowledging the client’s ISN.

Third handshake : client acknowledges the server’s SYN, and both sides enter ESTABLISHED .

2.2 Why Three Steps?

TCP is full‑duplex; both directions must be confirmed. With only two steps, a delayed SYN could be mistaken for a new connection, wasting resources. The third ACK guarantees that the server’s SYN was received before the connection is considered established.

2.3 ISN Randomness

Random ISN prevents sequence‑prediction attacks. To view current ISNs:

# View current TCP sequence numbers
cat /proc/net/tcp
cat /proc/net/tcp6

# Detailed view with ss
ss -ti state established

# Classic netstat view
netstat -tn | grep ESTABLISHED

2.4 MSS and Window Scaling

MSS = MTU – IP header – TCP header (typical MTU 1500 → MSS 1460).

MSS = MTU - IP header - TCP header
Typical values: MTU=1500, MSS=1460

When the receive window exceeds 65535 bytes, RFC 1323 defines a scaling factor.

# View window scaling factor
cat /proc/sys/net/ipv4/tcp_window_scaling

# View buffer settings
cat /proc/sys/net/ipv4/rmem
cat /proc/sys/net/ipv4/wmem

# Temporarily enable scaling
sysctl -w net.ipv4.tcp_window_scaling=1

3. Four‑Way Termination

3.1 Termination Process

Client                              Server
    |                               |
    |========= ESTABLISHED =========|
    |                               |
    |--- FIN=1, Seq=u ------------> |   (first FIN)
    |<--- ACK=1, Ack=u+1 ----------- |
    FIN_WAIT_1                CLOSE_WAIT
    |                               |
    |<--- FIN=1, Seq=v ------------ |   (second FIN)
    |--- ACK=1, Ack=v+1 ----------> |   (ACK to server FIN)
    TIME_WAIT                LAST_ACK
    |                               |
    |--- ACK=1 (final) ----------> |
    CLOSED                     CLOSED

First FIN : active closer enters FIN_WAIT_1 .

Second FIN : passive closer replies with ACK, moves to CLOSE_WAIT , then sends its own FIN and enters LAST_ACK .

Final ACK : active closer acknowledges, enters TIME_WAIT to ensure the ACK is received.

3.2 TIME_WAIT Details

TIME_WAIT lasts for 2 MSL (Maximum Segment Lifetime), typically 60 seconds (configurable via net.ipv4.tcp_fin_timeout). Its purposes:

Guarantee the final ACK reaches the passive side; if the ACK is lost, the passive side will retransmit FIN, which the active side can still accept.

Allow old packets to expire from the network, preventing them from interfering with new connections.

3.3 Kernel Parameters for TIME_WAIT

# View TIME_WAIT timeout (seconds)
cat /proc/sys/net/ipv4/tcp_fin_timeout   # default 60

# Count current TIME_WAIT sockets
ss -tan state time-wait | wc -l

3.4 Handling Excessive TIME_WAIT

Enable reuse: sysctl -w net.ipv4.tcp_tw_reuse=1 (requires timestamps enabled).

Reduce FIN timeout: sysctl -w net.ipv4.tcp_fin_timeout=30.

Increase bucket limit: sysctl -w net.ipv4.tcp_max_tw_buckets=100000.

Prefer keep‑alive or HTTP/2 for long‑lived connections.

4. RST Packets

RST indicates an abnormal termination that does not require acknowledgment.

Target port not listening – kernel replies with RST.

SO_LINGER set to 0 – close() sends RST immediately.

Unrecognised packet (e.g., sequence out of window).

Application crash – kernel may send FIN then RST if data remains.

4.1 Capturing RST

# Simulate connection to an unopened port
nc -zv 127.0.0.1 9999   # shows "Connection refused" (RST received)

# Capture RST packets
tcpdump -i eth0 'tcp[tcpflags] == tcp-rst'

# Count RST packets
netstat -s | grep -i "reset"

5. Connection Queues

5.1 Half‑Open (SYN) Queue

After a SYN arrives, the server moves the socket to SYN_RCVD and stores it in the SYN queue.

# Max half‑open queue length
cat /proc/sys/net/ipv4/tcp_max_syn_backlog   # default 128 (subject to somaxconn)

# Current SYN queue usage
ss -ltn state syn-recv

# Detect SYN flood
netstat -s | grep -i "SYN"
ss -ltn state syn-recv | wc -l

5.2 Full‑Open (Accept) Queue

When the three‑way handshake completes, the socket moves to the accept queue, awaiting accept() by the application.

# Max accept queue length (listen backlog)
cat /proc/sys/net/core/somaxconn   # default 128

# Current accept queue usage
ss -ltn state listen

# Observe Recv‑Q (used slots) and Send‑Q (backlog limit)
ss -ltn

5.3 Queue Size Recommendations

Set tcp_max_syn_backlog ≥ 2048.

Set somaxconn ≥ 2048.

Configure application listen backlog ≥ 1024 (e.g., Nginx backlog=65535, Go Listen(...) uses system somaxconn, Tomcat acceptCount=1000, Python listen(2048)).

6. Common Production Issues and Diagnosis

6.1 Immediate Disconnect (RST)

Possible causes: port not listening, full listen queue, firewall block, server crash.

Diagnosis steps:

Check listening state: ss -tlnp | grep 8080.

Inspect firewall rules: iptables -L -n or firewall-cmd --list-all.

View TCP states: ss -ti state established.

Capture packets: tcpdump -i eth0 port 8080 -nn.

Verify process status: systemctl status nginx / ps aux | grep nginx.

6.2 Excessive CLOSE_WAIT

Root cause: application fails to close sockets (e.g., missing close(), connection‑pool leaks, exception paths).

Diagnosis:

List CLOSE_WAIT sockets: ss -tup state close-wait.

Identify owning processes: ss -tup | grep CLOSE_WAIT.

Inspect code (Java jstack, Python strace, Go pprof).

Check keepalive and timeout settings.

Remediation: ensure proper close(), set reasonable SO_TIMEOUT, enable TCP keepalive, configure connection pools correctly.

6.3 Excessive TIME_WAIT

Typical cause: many short‑lived connections.

Diagnosis:

Overall state count: ss -s.

Detail TIME_WAIT sockets: ss -tan state time-wait.

Identify source IPs:

ss -tan state time-wait | awk '{print $4}' | cut -d: -f1 | sort | uniq -c | sort -rn

.

Remediation: enable tcp_tw_reuse, reduce tcp_fin_timeout, increase tcp_max_tw_buckets, use keep‑alive or HTTP/2, reuse connections on the client side.

6.4 Port Exhaustion

Symptoms: "Cannot assign requested address" or "Address already in use".

Cause: many short connections consume the ephemeral port range (default 32768‑60999).

Diagnosis:

Check current range: cat /proc/sys/net/ipv4/ip_local_port_range.

Count used ports: ss -tan | awk '{print $4}' | grep -E ':[0-9]+$' | wc -l.

Remediation: enlarge port range, enable tcp_tw_reuse, lower tcp_fin_timeout, prefer long‑lived connections.

6.5 Half‑Open (Half‑Open) Connections

Cause: one side crashes, the other does not detect the failure (no heartbeat).

Solution: enable TCP keepalive and tune its parameters.

# View current keepalive settings
cat /proc/sys/net/ipv4/tcp_keepalive_time   # default 7200s
cat /proc/sys/net/ipv4/tcp_keepalive_intvl  # default 75s
cat /proc/sys/net/ipv4/tcp_keepalive_probes # default 9

# Adjust for faster detection
sysctl -w net.ipv4.tcp_keepalive_time=600   # 10 min
sysctl -w net.ipv4.tcp_keepalive_intvl=30  # 30 s interval
sysctl -w net.ipv4.tcp_keepalive_probes=5  # fail after 5 probes

7. Monitoring Scripts

7.1 TCP Status Monitor

#!/bin/bash
# tcp_status_monitor.sh – monitor TCP connection states and alert on anomalies

echo "=== TCP Connection State Statistics ==="
ss -tan | awk '{print $1}' | sort | uniq -c | sort -rn

echo "
=== TIME_WAIT Connections ==="
timewait_count=$(ss -tan state time-wait | wc -l)
echo "Current TIME_WAIT: $timewait_count"
if [ $timewait_count -gt 50000 ]; then
  echo "Warning: TIME_WAIT count is high"
fi

echo "
=== CLOSE_WAIT Connections ==="
closewait_count=$(ss -tan state close-wait | wc -l)
echo "Current CLOSE_WAIT: $closewait_count"
if [ $closewait_count -gt 1000 ]; then
  echo "Warning: CLOSE_WAIT count is abnormal"
fi

echo "
=== SYN_RECVD (Half‑Open) Connections ==="
synrecv_count=$(ss -tan state syn-recv | wc -l)
echo "Current SYN_RECVD: $synrecv_count"
if [ $synrecv_count -gt 1000 ]; then
  echo "Warning: SYN queue may be under attack"
fi

echo "
=== Top 10 Established Sources ==="
ss -tan state established | awk '{print $4}' | cut -d: -f1 | sort | uniq -c | sort -rn | head -10

echo "
=== Latest TCP Error Statistics ==="
cat /proc/net/netstat | grep TcpExt | tail -1

7.2 Port Connection Monitor

#!/bin/bash
# port_conn_monitor.sh – monitor connections on a specific port
PORT=${1:-80}
THRESHOLD=${2:-1000}

echo "=== Port $PORT Connection Summary ==="
ss -tunlp | grep ":$PORT " | head -5

echo "
=== Established Connections on Port $PORT ==="
est_count=$(ss -tan "sport = :$PORT or dport = :$PORT" state established | wc -l)
echo "Current Established: $est_count"
if [ $est_count -gt $THRESHOLD ]; then
  echo "Warning: connection count exceeds $THRESHOLD"
fi

echo "
=== Per‑State Statistics for Port $PORT ==="
ss -tan "sport = :$PORT or dport = :$PORT" | awk 'NR>1 {print $1}' | sort | uniq -c | sort -rn

7.3 TCP Debug Script

#!/bin/bash
# tcp_debug.sh – capture traffic and analyze handshake/termination
OUTPUT_DIR="/tmp/tcpdump"
mkdir -p $OUTPUT_DIR
TIMESTAMP=$(date +%Y%m%d_%H%M%S)

# Capture for 60 seconds on port 8080
echo "Starting capture to $OUTPUT_DIR/tcp_$TIMESTAMP.pcap"
timeout 60 tcpdump -i eth0 -w "$OUTPUT_DIR/tcp_${TIMESTAMP}.pcap" port 8080 2>/dev/null &

sleep 10
echo "=== Status after 10 s ==="
ss -tunlp | grep 8080

sleep 10
echo "=== Status after 20 s ==="
ss -tunlp | grep 8080

wait

echo "
=== Handshake Packet Count ==="
tcpdump -r "$OUTPUT_DIR/tcp_${TIMESTAMP}.pcap" 2>/dev/null | grep -E "(SYN|ACK|FIN|RST)" | wc -l

echo "
=== RST Packet Count ==="
tcpdump -r "$OUTPUT_DIR/tcp_${TIMESTAMP}.pcap" 2>/dev/null | grep -c "RST"
echo " RST packets"

8. Summary and Tuning Recommendations

The three‑way handshake guarantees bidirectional confirmation, while the four‑way termination safely closes both directions and uses TIME_WAIT to prevent old packets from interfering with new connections. Key states to monitor are TIME_WAIT, CLOSE_WAIT, SYN_RCVD, and ESTABLISHED. Common troubleshooting commands include ss, netstat, and tcpdump. For production environments, the following sysctl settings are recommended:

# /etc/sysctl.conf additions
net.ipv4.tcp_tw_reuse = 1          # allow TIME_WAIT reuse
net.ipv4.tcp_fin_timeout = 30      # shorten FIN timeout
net.ipv4.tcp_max_tw_buckets = 100000
net.ipv4.tcp_max_syn_backlog = 2048
net.core.somaxconn = 2048
net.ipv4.ip_local_port_range = 1024 65535
net.ipv4.tcp_keepalive_time = 600
net.ipv4.tcp_keepalive_intvl = 30
net.ipv4.tcp_keepalive_probes = 5
net.ipv4.tcp_timestamps = 1        # required for tw_reuse

# Apply changes
sysctl -p

By combining a solid grasp of TCP state mechanics with systematic use of the commands and scripts above, engineers can quickly pinpoint the root cause of connection issues and apply appropriate kernel or application‑level adjustments.