Master TCP Handshakes and Teardowns: Deep Dive with Wireshark and Linux Tools

TCP Header Overview

The TCP header is 20 bytes minimum and consists of source/destination ports, sequence and acknowledgment numbers, data offset, flags, window size, checksum, urgent pointer and optional fields.

0               1               2               3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|          Source Port          |       Destination Port      |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                        Sequence Number                     |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                    Acknowledgment Number                   |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|  Data |    |C|E|U|A|P|R|S|F|                               |Window |
| Offset| Flags |W|C|R|C|S|S|Y|I|            Window Size       |
|       |       |R|E|G|K|H|T|N|N|                               |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|          Checksum           |        Urgent Pointer        |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                    Options and Padding                     |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                             Data                            |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

Source Port (16 bit): client‑chosen random port

Destination Port (16 bit): server listening port

Sequence Number (32 bit): identifies the byte stream

Acknowledgment Number (32 bit): next expected byte

Data Offset (4 bit): header length in 32‑bit words

Flags (9 bit): control bits (CWR, ECE, URG, ACK, PSH, RST, SYN, FIN, NS)

Window (16 bit): advertised receive window size

Checksum (16 bit): header + data integrity

Urgent Pointer (16 bit): points to urgent data

TCP Flags

Each flag has a specific meaning. Common combinations are shown by tcpdump.

# TCP flags (9 total)
# - URG: urgent pointer valid
# - ACK: acknowledgment field valid
# - PSH: push data to application layer
# - RST: reset connection
# - SYN: synchronize sequence numbers (connection setup)
# - FIN: finish, close connection
# - ECE: ECN‑Echo (explicit congestion notification)
# - CWR: congestion window reduced
# - NS: nonce sum (reserved)

# tcpdump flag combinations
# [S]   = SYN
# [S.]  = SYN+ACK
# [.]   = ACK
# [F]   = FIN
# [F.]  = FIN+ACK
# [R]   = RST
# [R.]  = RST+ACK
# [P.]  = PSH+ACK
# [S.] [.] [R] = three‑way handshake sequence

Capture specific flag sets with tcpdump:

# Capture all packets with the RST flag
sudo tcpdump -i eth0 'tcp[tcpflags] & tcp-rst != 0' -n

# Capture packets that carry PSH+ACK
sudo tcpdump -i eth0 'tcp[tcpflags] == tcp-ack and tcp[tcpflags] & tcp-psh != 0' -n

Sequence and Acknowledgment Mechanism

The sender increments the sequence number by the number of bytes transmitted; the receiver acknowledges the next expected byte.

class TCPSeqAck:
    def __init__(self, isn):
        # ISN: Initial Sequence Number
        self.isn = isn
        self.next_expected = isn + 1

    def send_data(self, data):
        """Send data and return the current sequence number"""
        seq = self.next_expected
        self.next_expected += len(data)
        return seq

    def receive_ack(self, ack):
        """Process a received ACK"""
        if ack > self.next_expected:
            return "ACK is larger than expected"
        elif ack == self.next_expected:
            return "Full ACK – all data received"
        else:
            return "Partial ACK"

# Demo
tcp = TCPSeqAck(isn=1000)
seq1 = tcp.send_data(b"Hello")  # seq=1000, 5 bytes sent
print(f"Sent data, sequence: {seq1}, next expected: {tcp.next_expected}")
result = tcp.receive_ack(1005)   # ACK 1005
print(f"Received ACK 1005: {result}")

Three‑Way Handshake

The handshake guarantees that both ends can send and receive, negotiates the Initial Sequence Number (ISN), and protects against stale SYN packets.

Client sends SYN with its ISN.

Server replies with SYN+ACK, containing its own ISN and acknowledging the client’s ISN+1.

Client sends ACK, acknowledging the server’s ISN+1. The connection enters ESTABLISHED state.

Detailed state transition (client side):

Timeline          Client                         Server
  |                     |                               |
  | 1. CLOSED          | 1. LISTEN                     |
  | --> Choose client ISN (c_isn)               |
  | --> Send SYN                               |
T1|-------------------------------------------->|   SEQ=c_isn, Flags=[SYN], state=SYN_SENT
  |                     |                               |
  |                     | 2. Receive SYN, choose s_isn    |
  |                     | --> Send SYN+ACK               |
T2|<--------------------------------------------|   SEQ=s_isn, ACK=c_isn+1, Flags=[SYN,ACK], state=SYN_RECEIVED
  |                     |                               |
  | 3. Receive SYN+ACK                         |
  | --> Verify ACK, send ACK                  |
T3|-------------------------------------------->|   SEQ=c_isn+1, ACK=s_isn+1, Flags=[ACK], state=ESTABLISHED
  |                     |                               |
  |                     | 4. ACK received, connection established |

Capture the handshake with tcpdump and analyse in Wireshark:

# Capture SYN packets on port 80
sudo tcpdump -i eth0 -w /tmp/handshake.pcap 'tcp port 80 and tcp[tcpflags] & tcp-syn != 0'
# In another terminal, issue a request
curl -I http://example.com
# Open the capture in Wireshark
wireshark /tmp/handshake.pcap &
# Or use tshark for a quick view
tshark -r /tmp/handshake.pcap -Y "tcp.flags.syn==1 or tcp.flags.synack==1 or tcp.flags.ack==1"

Four‑Way Teardown

Because TCP is full‑duplex, each side must close its sending direction independently, resulting in four packets (FIN, ACK, FIN, ACK).

Active closer sends FIN – “I will not send more data”.

Passive side ACKs the FIN but may still have data to send.

After the passive side finishes sending, it sends its own FIN.

The active side ACKs the second FIN and enters TIME_WAIT to ensure the final ACK is received.

State transition (client side):

Timeline          Client                         Server
  |                     |                               |
  | Assume state: ESTABLISHED                |
  | Application calls close()                |
  | --> Send FIN, enter FIN_WAIT_1           |
T1|----------------------------------------->|   SEQ=1000, ACK=2000, Flags=[FIN,ACK], state=FIN_WAIT_1
  |                     |                               |
  |                     | 2. Receive FIN, send ACK        |
T2|<-----------------------------------------|   SEQ=2000, ACK=1001, Flags=[ACK], state=CLOSE_WAIT
  |                     |                               |
  | 3. Receive ACK       |                               |
  | --> Enter FIN_WAIT_2 |                               |
  |                     |                               |
  | 4. Application finishes data, calls close() |
  | --> Send FIN         |                               |
T3|<-----------------------------------------|   SEQ=2000, ACK=1001, Flags=[FIN,ACK], state=LAST_ACK
  |                     |                               |
  | 5. Receive FIN       |                               |
  | --> Send ACK         |                               |
T4|----------------------------------------->|   SEQ=1001, ACK=2001, Flags=[ACK], state=TIME_WAIT
  |                     |                               |
  | Wait 2MSL, then CLOSED                |

TIME_WAIT ensures the final ACK reaches the peer and that delayed packets from the old connection are discarded.

# Default Linux TIME_WAIT timeout (2 MSL) is 60 seconds
cat /proc/sys/net/ipv4/tcp_fin_timeout   # 60
# Reduce timeout (use with caution)
echo 30 > /proc/sys/net/ipv4/tcp_fin_timeout
# Enable reuse of TIME_WAIT sockets (client side)
echo 1 > /proc/sys/net/ipv4/tcp_tw_reuse

CLOSE_WAIT problems occur when the server receives a FIN, ACKs it, but the application never calls close(). Diagnose with:

# List connections stuck in CLOSE_WAIT
netstat -an | grep CLOSE_WAIT | head -20
# Find owning process
ss -tlnp | grep :3306

TCP State Diagram

The following ASCII diagram shows the complete TCP state machine.

Application calls
                |
                v
          +-----------+
          |  CLOSED   |
          +-----------+
                |
                | passive open (listen)
                v
          +-----------+        +-----------+
          |   LISTEN  |<-------| SYN_SENT  |<-------+
          +-----------+        +-----------+        |
                |                     |            |
                |   send SYN          |   receive SYN
                v                     v            |
          +-----------+        +-----------+    |
          | SYN_RCVD  |<-------| ESTABLISHED|    |
          +-----------+        +-----------+    |
                |                     ^            |
                |   receive SYN+ACK   |   send ACK |
                v                     |            |
          +---------------------------+
          |      ESTABLISHED          |
          +---------------------------+
                |                     ^
                |                     |
                | active close        | passive close
                | send FIN            | receive FIN
                v                     |
          +-----------+               |
          |FIN_WAIT_1 |               |
          +-----------+               |
                |                     |
                | receive ACK         | receive FIN
                | (half‑close)         | send ACK
                v                     |
          +-----------+               |
          |FIN_WAIT_2 |<--------------+
          +-----------+       receive ACK
                |
                | receive FIN
                | send ACK
                v
          +-----------+
          | TIME_WAIT |
          +-----------+
                |
                | 2 MSL timeout
                v
          +-----------+
          |  CLOSED   |
          +-----------+

Server side: LISTEN → SYN_RCVD → ESTABLISHED → CLOSE_WAIT → LAST_ACK → CLOSED
Client side: CLOSED → SYN_SENT → ESTABLISHED → FIN_WAIT_1 → FIN_WAIT_2 → TIME_WAIT → CLOSED

Connection Management in Practice

Backlog Queues

Two queues are used by a listening server:

Half‑open (SYN) queue : holds incoming SYNs until the three‑way handshake completes. Size is controlled by tcp_max_syn_backlog (default 128).

Full‑open (accept) queue : holds fully established connections waiting for accept(). Size is set by the listen() backlog argument and the kernel limit somaxconn.

# View current SYN backlog limit
cat /proc/sys/net/ipv4/tcp_max_syn_backlog

# View maximum accept queue size
cat /proc/sys/net/core/somaxconn

# Increase limits (temporary)
echo 2048 > /proc/sys/net/ipv4/tcp_max_syn_backlog
echo 1024 > /proc/sys/net/core/somaxconn

# Permanent change – add to /etc/sysctl.conf
cat >> /etc/sysctl.conf <<'EOF'
net.core.somaxconn = 1024
net.ipv4.tcp_max_syn_backlog = 2048
EOF
sysctl -p

TCP Keepalive

Keepalive probes detect dead idle connections, useful for long‑lived services.

# System‑wide keepalive parameters (seconds)
# Enable keepalive probes
echo 1 > /proc/sys/net/ipv4/tcp_keepalive_probes
# Idle time before first probe (default 7200)
echo 7200 > /proc/sys/net/ipv4/tcp_keepalive_time
# Interval between probes (default 75)
echo 75 > /proc/sys/net/ipv4/tcp_keepalive_intvl
# Number of probes before declaring the connection dead
echo 9 > /proc/sys/net/ipv4/tcp_keepalive_probes

# Enable keepalive in a Python socket
import socket
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.setsockopt(socket.SOL_SOCKET, socket.SO_KEEPALIVE, 1)
sock.setsockopt(socket.IPPROTO_TCP, socket.TCP_KEEPIDLE, 7200)
sock.setsockopt(socket.IPPROTO_TCP, socket.TCP_KEEPINTVL, 75)
sock.setsockopt(socket.IPPROTO_TCP, socket.TCP_KEEPALIVE, 9)

Performance Optimizations

Enable optional TCP features for better throughput and reliability.

# Enable TCP timestamps (required for PAWS)
cat /proc/sys/net/ipv4/tcp_timestamps   # 1 = enabled

# Enable selective acknowledgments (SACK)
cat /proc/sys/net/ipv4/tcp_sack          # 1 = enabled

# Enable window scaling for large windows
cat /proc/sys/net/ipv4/tcp_window_scaling   # 1 = enabled

# View current window scaling factors for a connection
ss -ti

# Adjust MTU and MSS (Maximum Segment Size)
# Typical Ethernet MTU = 1500 → MSS = 1460
# Verify MSS in captured packets or with ss -i
ss -i | grep -E "rcv_space|snd_wnd"

Common TCP Troubleshooting

Connection Timeout

Typical steps to diagnose a timeout:

# 1. Verify basic network reachability
ping -c 5 target_host

# 2. Check routing path
traceroute target_host
# or mtr target_host

# 3. Confirm the destination port is open
nc -zv target_host 80
# or nmap -p 80 target_host

# 4. Inspect local port range (default 32768‑60999)
cat /proc/sys/net/ipv4/ip_local_port_range

# 5. Capture the handshake for analysis
sudo tcpdump -i eth0 host target_host and port 80 -w /tmp/timeout.pcap

# 6. Analyse the SYN/ACK exchange
tshark -r /tmp/timeout.pcap -Y "tcp.flags.syn==1" -T fields \
    -e frame.time -e ip.src -e tcp.srcport -e ip.dst -e tcp.dstport

Connection Reset (RST)

RST packets indicate an abrupt termination. Capture them to find the cause.

# Capture RST packets
sudo tcpdump -i eth0 'tcp[tcpflags] & tcp-rst != 0' -n

# Common reasons:
# • Destination port not listening
# • SO_LINGER set to 0 (abort)
# • Server crash or restart
# • Firewall rejecting the packet

Queue Full

When the SYN or accept queue overflows, new connections are dropped.

# Check for SYN queue overflow counters
netstat -s | grep -i "SYN"
# Example counters: TCPRcvCoalesce, TCPOFODrop

# Check accept queue length (Recv‑Q) for a listening socket
ss -ltn | grep :80

# Increase limits (see Backlog Queues section) or tune the application to accept connections faster.

Excessive TIME_WAIT

Large numbers of TIME_WAIT sockets can exhaust kernel resources under high connection churn.

# Count TIME_WAIT sockets
ss -ant | awk '{print $1}' | sort | uniq -c | sort -rn

# Reduce timeout (cautiously)
echo 30 > /proc/sys/net/ipv4/tcp_fin_timeout

# Enable reuse of TIME_WAIT sockets (client side)
echo 1 > /proc/sys/net/ipv4/tcp_tw_reuse

# Force immediate close with SO_LINGER (generates RST and may drop data)
struct linger ling;
ling.l_onoff = 1;
ling.l_linger = 0;
setsockopt(sock, SOL_SOCKET, SO_LINGER, &ling, sizeof(ling));
close(sock);

Wireshark Advanced Analysis

Common Filter Expressions

# Basic filters
tcp.port == 80                # port 80
tcp.srcport == 12345          # source port 12345
tcp.dstport == 443            # destination port 443
ip.addr == 192.168.1.100      # any packet to/from this IP
tcp.flags.syn == 1            # SYN packets only
tcp.flags.ack == 1            # ACK packets only
tcp.flags.fin == 1            # FIN packets only
tcp.flags.rst == 1            # RST packets only

# Combined filters
tcp.port == 80 and ip.addr == 192.168.1.100
tcp.flags.syn == 1 and tcp.flags.ack == 0   # pure SYN
tcp.flags.syn == 1 and tcp.flags.ack == 1   # SYN+ACK

# Sequence/ACK filters
tcp.seq == 1000
tcp.ack == 2000

# Time filter (relative time < 1 s)
frame.time_relative < 1

# Expert info filters
tcp.analysis.retransmission
tcp.analysis.duplicate_ack
tcp.analysis.out_of_order
tcp.analysis.fast_retransmission

Follow TCP Stream

# Find the stream index
tshark -r capture.pcap -q -z "conv,tcp" | head -20
# Export the raw data of a specific stream
tshark -r capture.pcap -Y "tcp.stream eq 0" -T fields -e data
# Export an HTTP session
tshark -r capture.pcap -Y "http" -T fields -e ip.src -e http.request.method -e http.request.uri

TCP Statistics

# Flow statistics (0.1 s intervals)
tshark -r capture.pcap -q -z "io,stat,0.1,tcp.len"

# Connection statistics
tshark -r capture.pcap -q -z "conv,tcp"

# Retransmission statistics
tshark -r capture.pcap -q -z "io,stat,0.1,tcp.analysis.retransmission"

# Graphs in Wireshark → Statistics → TCP Stream Graphs:
# • Time‑Sequence (Stevens)
# • Throughput
# • Round‑Trip Time

Real‑World Case Studies

Case 1 – Intermittent Web Service Connection Failures

Symptom : Users occasionally receive connection errors; a page refresh resolves the issue.

Investigation :

# Capture traffic on the server
sudo tcpdump -i eth0 -w /tmp/http_issue.pcap 'tcp port 80' &
sleep 30
kill %1

# Look for SYN retransmissions (indicates backlog overflow)
tshark -r /tmp/http_issue.pcap -Y "tcp.analysis.retransmission" | wc -l

# Check the listen backlog
ss -ltn | grep :80   # Recv‑Q should be near 0, Send‑Q small

# Verify kernel limits
cat /proc/sys/net/core/somaxconn
cat /proc/sys/net/ipv4/tcp_max_syn_backlog

Root cause : The server’s somaxconn and SYN backlog were too low, causing SYN queue overflow during traffic spikes.

Solution :

# Increase kernel limits
echo 4096 > /proc/sys/net/core/somaxconn
echo 8192 > /proc/sys/net/ipv4/tcp_max_syn_backlog

# Example Nginx configuration
# worker_processes auto;
# worker_connections 4096;
# listen 80 backlog=4096;

Case 2 – Database Connection Pool Exhaustion

Symptom : Application logs show “Too many connections”.

# Show MySQL connection count
mysql -u root -p -e "SHOW PROCESSLIST;" | wc -l
mysql -u root -p -e "SHOW STATUS LIKE 'Threads_connected';"

# Check TIME_WAIT sockets (may indicate short‑lived connections)
netstat -an | grep TIME_WAIT | wc -l

# Identify which hosts open the most connections
netstat -ant | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -rn | head -10

Root cause : The application opened a new TCP connection for every request, exhausting the MySQL connection limit under load.

Solution : Use a connection pool.

import mysql.connector.pooling

pool = mysql.connector.pooling.MySQLConnectionPool(
    pool_name="mypool",
    pool_size=10,
    host="localhost",
    database="test"
)
conn = pool.get_connection()
cursor = conn.cursor()
# ... use the connection ...
conn.close()  # returns the connection to the pool

Case 3 – API Request Timeouts on Mobile Clients

Symptom : Mobile API calls frequently time out.

# Test network latency and jitter
ping -c 20 api.example.com

# Capture the TCP traffic
sudo tcpdump -i eth0 -w /tmp/api_tcp.pcap host api.example.com &
# Reproduce the timeout, then stop capture

# Count retransmissions (indicates packet loss)
tshark -r /tmp/api_tcp.pcap -Y "tcp.analysis.retransmission" | wc -l

# In Wireshark, inspect the TCP window size; a zero window means the receiver’s buffer is full.

Root cause : Server processing was slow, causing the receive window to shrink to zero; the client timed out waiting for data.

Solution :

# Increase kernel socket buffers
echo 16777216 > /proc/sys/net/core/rmem_max
echo 16777216 > /proc/sys/net/core/wmem_max

# Nginx buffer tuning (example)
# proxy_buffering on;
# proxy_buffer_size 128k;
# proxy_buffers 4 256k;

# Application‑level optimizations:
# • Increase worker threads
# • Use asynchronous processing
# • Optimize database queries

Quick Reference

Three‑Way Handshake State Machine

Client                              Server
CLOSED ─────────────────────► LISTEN
   |                                 |
   | 1. Send SYN (SEQ=c_isn)          |
   └─────────────────────────────────► SYN_SENT
   |                                 |
   |               ◄───────────────── SYN+ACK (SEQ=s_isn, ACK=c_isn+1)
   |                                 |
   └─────────────────────────────────► ACK (ACK=s_isn+1)
   |                                 |
   └─────────────────────────────────► ESTABLISHED (both sides)

Four‑Way Teardown State Machine

Active                              Passive
ESTABLISHED ─────────────────────► ESTABLISHED
   |                                 |
   | 1. Send FIN (SEQ=x)              |
   └─────────────────────────────────► FIN_WAIT_1
   |                                 |
   |               ◄───────────────── ACK (ACK=x+1)
   |                                 |
   └─────────────────────────────────► FIN_WAIT_2
   |                                 |
   |               ◄───────────────── FIN (SEQ=y)
   |                                 |
   └─────────────────────────────────► TIME_WAIT (after ACK)
   |                                 |
   └─────────────────────────────────► CLOSED (after 2 MSL)

TCP State Quick Lookup

LISTEN : server waiting for connections

SYN_SENT : client has sent SYN

SYN_RCVD : server received SYN

ESTABLISHED : connection active (both sides)

FIN_WAIT_1 : active closer sent FIN

FIN_WAIT_2 : ACK for our FIN received, waiting for peer FIN

CLOSE_WAIT : passive side received FIN, waiting for application close()

TIME_WAIT : waiting 2 MSL after final ACK

LAST_ACK : passive side sent FIN, awaiting final ACK

Common Command Cheat Sheet

# List all TCP connections
ss -ant

# Show detailed timers and state information
ss -ti

# List listening sockets with owning process
ss -tlnp

# Capture packets to a file
tcpdump -i eth0 'tcp port 80' -w a.pcap
# Analyse with tshark
tshark -r a.pcap -Y "tcp" -T fields

# Show kernel TCP parameters
cat /proc/sys/net/ipv4/tcp_*   # or sysctl -a | grep tcp

# Show TCP statistics
netstat -s | grep -i tcp
ss -s

Troubleshooting Flowchart

TCP connection issue
    |
    ├─ Handshake problems
    │   ├─ SYN not sent → check network
    │   ├─ SYN not received → check firewall
    │   └─ SYN+ACK missing → capture packets
    |
    ├─ Teardown problems
    │   ├─ FIN not sent → verify application close()
    │   ├─ Excessive TIME_WAIT → tune tcp_fin_timeout or enable reuse
    │   └─ CLOSE_WAIT high → ensure application closes sockets
    |
    ├─ Connection reset
    │   ├─ RST source → capture and analyse
    │   └─ Frequent retransmissions → check network quality
    |
    └─ Performance issues
        ├─ Window size zero → increase buffer sizes
        ├─ Queue full → raise somaxconn / tcp_max_syn_backlog
        └─ Many short connections → use connection pooling