Master TCP Handshakes and Teardowns with Wireshark: A Complete Interview Guide

Overview

This guide explains the TCP three‑way handshake and four‑way teardown at the packet level, shows how to capture and analyse them with Wireshark or tcpdump, and covers related production‑level topics such as retransmission, congestion control, TIME_WAIT optimisation and kernel tuning.

TCP Header Layout

The TCP header is 20 bytes minimum and up to 60 bytes with options. The most relevant fields for connection management are:

0               1               2               3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|        Source Port          |     Destination Port          |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                        Sequence Number                        |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                    Acknowledgment Number                      |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Data |   |C|E|U|A|P|R|S|F|               Window                    |
|Offset| Rsrvd|W|C|R|C|S|S|Y|I|                                         |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|          Checksum           |        Urgent Pointer          |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                          Options (variable)                    |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

Key fields:

Sequence Number – 32‑bit number of the first byte in the segment.

Acknowledgment Number – 32‑bit number of the next byte the sender expects.

Flags – six bits (SYN, ACK, FIN, RST, PSH, URG) that drive handshake and teardown.

TCP Flags

SYN – synchronise sequence numbers, initiates a connection.

ACK – acknowledges received data; valid only when ACK=1.

FIN – request graceful close of one direction.

RST – abort the connection immediately.

PSH – push data to the receiving application.

URG – urgent pointer is valid.

Three‑Way Handshake

The handshake exchanges initial sequence numbers (ISN) and confirms both sides' send/receive capabilities.

Client                               Server
|                                   |
|--- SYN, Seq=x ------------------->|
|   (SYN_SENT)          (SYN_RCVD) |
|<--- SYN+ACK, Seq=y, Ack=x+1 -----|
|                                   |
|--- ACK, Seq=x+1, Ack=y+1 ------->|
|   (ESTABLISHED)   (ESTABLISHED) |

Step‑by‑step packet details:

First packet (client) : SYN=1, random ISN Seq=x, no ACK, typical options (MSS, WS, SACK_PERM, TSval).

Second packet (server) : SYN=1, ACK=1, its own ISN Seq=y, Ack=x+1, same options.

Third packet (client) : ACK=1, Seq=x+1, Ack=y+1. Both sides move to ESTABLISHED .

Wireshark shows relative sequence numbers by default; disable "Relative sequence numbers" in Preferences → Protocols → TCP to see the real 32‑bit ISN.

Why Three Packets?

Prevents stale SYN packets from establishing a connection that the client never saw.

Both sides must confirm four capabilities (client send, client receive, server send, server receive); the three packets provide exactly those confirmations.

Four‑Way Teardown

Active closer                     Passive closer
|                                 |
|--- FIN, Seq=u ----------------->|
|   (FIN_WAIT_1)          (CLOSE_WAIT) |
|<--- ACK, Ack=u+1 ----------------|
|   (FIN_WAIT_2)                 |
|                                 |
|<--- FIN, Seq=w, Ack=u+1 ---------|
|   (LAST_ACK)               (TIME_WAIT) |
|--- ACK, Seq=u+1, Ack=w+1 ------->|
|                                 |
| (wait 2 MSL)                    |
|   (CLOSED)                      |

Key points:

The active side sends FIN (via close() or shutdown(SHUT_WR)).

The passive side ACKs the FIN, moves to CLOSE_WAIT , continues sending any remaining data.

When the passive side finishes, it sends its own FIN; the active side ACKs and enters TIME_WAIT .

Linux’s delayed ACK may merge the final ACK and FIN, reducing the exchange to three packets.

TIME_WAIT Details

TIME_WAIT lasts 2 MSL (60 s on Linux). It guarantees the final ACK reaches the peer and lets stray packets from the old connection expire before a new connection re‑uses the same 4‑tuple.

Each TIME_WAIT entry consumes ~0.25 KB kernel memory; thousands mainly cause port exhaustion.

Typical optimisation knobs:

# Reuse outbound TIME_WAIT ports (safe for client‑side only)
sysctl -w net.ipv4.tcp_tw_reuse=1
# Shorten FIN_WAIT_2 timeout
sysctl -w net.ipv4.tcp_fin_timeout=15
# Expand the ephemeral port range
sysctl -w net.ipv4.ip_local_port_range="1024 65535"
# Use connection pools (HTTP Keep‑Alive, gRPC, DB pools) to reduce short‑lived connections

Do NOT enable tcp_tw_recycle; it was removed after Linux 4.12 because it breaks NAT.

Wireshark Capture Practice

Capture Preparation

# Capture on the server and save to a pcap file
sudo tcpdump -i eth0 -s 0 -w /tmp/tcp-handshake.pcap port 443
# Generate traffic from another terminal
curl -v https://example.com
# Stop capture with Ctrl+C and open the pcap in Wireshark

Useful Display Filters

# Basic filters
tcp.port == 443               # source or destination port 443
ip.addr == 192.168.1.10       # any packet involving this IP

# Flag filters (handshake / teardown)
tcp.flags.syn == 1 && tcp.flags.ack == 0   # first SYN
tcp.flags.syn == 1 && tcp.flags.ack == 1   # SYN+ACK
tcp.flags.fin == 1                         # FIN packets
tcp.flags.reset == 1                       # RST packets

# Analysis filters
tcp.analysis.retransmission                # retransmitted segments
tcp.analysis.duplicate_ack                 # duplicate ACKs
tcp.analysis.zero_window                   # zero‑window events

Follow TCP Stream

Right‑click any packet → Follow → TCP Stream to view the complete lifecycle from SYN to FIN. This view is the quickest way to verify a single connection.

tcpdump Command‑Line Tips

# Capture only SYN packets (new connections)
sudo tcpdump -i any 'tcp[tcpflags] & tcp-syn != 0' -nn

# Capture only RST packets (abnormal termination)
sudo tcpdump -i any 'tcp[tcpflags] & tcp-rst != 0' -nn

# Capture SYN without ACK (first handshake)
sudo tcpdump -i any 'tcp[tcpflags] & (tcp-syn|tcp-ack) == tcp-syn' -nn

# Capture traffic between two hosts and write to a file (limit to 10 000 packets)
sudo tcpdump -i eth0 host 10.0.0.1 and port 443 -s 0 -w /tmp/debug.pcap -c 10000

# Show absolute sequence numbers and timestamps
sudo tcpdump -i any -nn -S -tttt port 80

TCP Retransmission Mechanisms

Timeout Retransmission (RTO)

After sending data, the sender starts a retransmission timer (RTO). If no ACK arrives before the timer expires, the segment is retransmitted. Linux computes RTO using Jacobson’s algorithm:

SRTT = (1‑α) * SRTT + α * RTT_sample   (α = 1/8)
RTTVAR = (1‑β) * RTTVAR + β * |SRTT‑RTT|   (β = 1/4)
RTO = SRTT + 4 * RTTVAR
# Minimum RTO = 200 ms (TCP_RTO_MIN), maximum = 120 s (TCP_RTO_MAX)
# On each timeout RTO doubles (exponential back‑off) until tcp_retries2 (default 15) is reached.

Fast Retransmit

When three duplicate ACKs are received, the sender immediately retransmits the missing segment without waiting for the timer.

Sender                     Receiver
|--- Seq=1, Len=1000 --->|
|--- Seq=1001, Len=1000 --->|   (Seq=1001 lost)
|<--- ACK=1001 (Dup) -----|   (duplicate ACK #1)
|--- Seq=2001, Len=1000 --->|
|<--- ACK=1001 (Dup) -----|   (duplicate ACK #2)
|--- Seq=3001, Len=1000 --->|
|<--- ACK=1001 (Dup) -----|   (duplicate ACK #3 triggers fast retransmit)
|--- Seq=1001, Len=1000 --->|   (re‑sent segment)
|<--- ACK=5001 ------------|   (SACK acknowledges all received data)

Wireshark marks fast‑retransmitted packets as [TCP Fast Retransmission] and duplicate ACKs as [TCP Dup ACK].

Selective Acknowledgment (SACK)

SACK lets the receiver inform the sender exactly which blocks of data were received, allowing the sender to retransmit only the missing segments.

# Verify SACK is enabled (default 1)
sysctl net.ipv4.tcp_sack
# In Wireshark filter: tcp.options.sack

Congestion Control: BBR vs Cubic

Cubic (Linux default)

Cubic is loss‑based. It grows the congestion window exponentially during slow start, then follows a cubic function after a loss event.

Pros: simple, works well in low‑latency data‑center networks.

Cons: slow recovery on long‑fat‑pipe or high‑loss links; can cause bufferbloat.

BBR (Bottleneck Bandwidth and RTT)

BBR probes the bottleneck bandwidth (BtlBw) and the minimum RTT (RTprop) and sets the sending rate to BtlBw, keeping in‑flight data at BtlBw × RTprop.

Pros: high bandwidth utilisation on high‑latency links, low queuing delay, tolerant to random loss.

Cons: fairness issues when co‑existing with Cubic (improved in BBR v3); less benefit on low‑latency LANs.

# Show current algorithm
sysctl net.ipv4.tcp_congestion_control
# Switch to BBR
sysctl -w net.ipv4.tcp_congestion_control=bbr
# Persist the change
cat >> /etc/sysctl.conf <<'EOF'
net.core.default_qdisc = fq
net.ipv4.tcp_congestion_control = bbr
EOF
sysctl -p

Best Practices

Connection Management

Server side: increase net.core.somaxconn and net.ipv4.tcp_max_syn_backlog, enable SYN cookies ( net.ipv4.tcp_syncookies=1).

Application layer: use HTTP/2 or gRPC multiplexing, configure reasonable Keep‑Alive timeouts, employ connection pools, and perform graceful shutdown with shutdown(SHUT_WR) followed by close().

Capture Analysis

Use BPF filters to limit capture size.

In production, capture with -c (packet count) or -G / -W (time‑based rotation) to avoid disk exhaustion.

Post‑capture: capinfos for file summary, Wireshark Statistics → Conversations for connection distribution, tshark for batch processing.

Common TCP Problems

Connection timeout : verify the server is listening, check that SYN packets leave the client (tcpdump), and inspect firewall rules.

RST packets : identify the sender (kernel, application, firewall, load balancer) and the cause (port not listening, abrupt close, timeout).

SYN flood : monitor ss -ant state syn-recv, enable SYN cookies, enlarge tcp_max_syn_backlog, and apply iptables rate‑limiting.

Full‑connection queue overflow : increase net.core.somaxconn and the application backlog; set tcp_abort_on_overflow=0 to avoid RST responses.

Kernel Tuning for Production

# /etc/sysctl.d/99-tcp-tuning.conf
# Connection queues
net.core.somaxconn = 65535
net.ipv4.tcp_max_syn_backlog = 65535

# TIME_WAIT optimisation
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_fin_timeout = 15
net.ipv4.tcp_max_tw_buckets = 262144
net.ipv4.ip_local_port_range = 1024 65535

# SYN flood defence
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_synack_retries = 2
net.ipv4.tcp_syn_retries = 3

# Keep‑Alive
net.ipv4.tcp_keepalive_time = 600
net.ipv4.tcp_keepalive_intvl = 15
net.ipv4.tcp_keepalive_probes = 5

# Buffer sizes
net.ipv4.tcp_rmem = 4096 87380 16777216
net.ipv4.tcp_wmem = 4096 65536 16777216
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216

# Congestion control
net.core.default_qdisc = fq
net.ipv4.tcp_congestion_control = bbr

# Optional features
net.ipv4.tcp_fastopen = 3
net.ipv4.tcp_sack = 1
net.ipv4.tcp_window_scaling = 1
net.ipv4.tcp_abort_on_overflow = 0

Apply with sysctl -p /etc/sysctl.d/99-tcp-tuning.conf and verify with

sysctl net.core.somaxconn net.ipv4.tcp_tw_reuse net.ipv4.tcp_congestion_control

.

Interview Quick Reference

Why three‑handshake? Prevents stale SYNs and confirms both sides' send/receive capabilities.

Why four‑handshake? TCP is full‑duplex; each direction must be closed independently.

Purpose of TIME_WAIT? Guarantees final ACK delivery and lets old packets expire before reusing the 4‑tuple.

How to defend against SYN flood? Enable SYN cookies, enlarge half‑connection queue, apply rate‑limiting.

Difference between RST and FIN? FIN is graceful (four‑step) close; RST aborts immediately.

BBR vs Cubic? Cubic is loss‑based; BBR probes bandwidth and RTT for higher throughput on high‑latency links.

What happens when accept queue is full? By default the kernel drops ACKs (client retries); setting tcp_abort_on_overflow=1 makes the server send RST.

References

RFC 9293 – TCP Specification (2022).

Google BBR Congestion Control research.

Wireshark TCP Analysis Documentation.

Linux Kernel Networking – TCP Implementation.

tcpdump Manual Page.