Linux Network Troubleshooting: tcpdump, netstat, ss Deep Dive with Case Studies

Why netstat, ss, and tcpdump Together

The trio forms the "iron triangle" of Linux network troubleshooting with clear division of labor: netstat: Shows all socket states, connection counts, listening ports, peer addresses. Provides a full view but slow under high connection counts. ss: Replacement for netstat, reads directly from /proc/net/tcp, 10-50x faster, more structured output. tcpdump: Captures packets at data link layer, revealing actual bytes on the wire — the only evidence that never lies.

They are often combined: use ss to spot anomalies, then tcpdump to capture the scene, finally netstat / ss to compare pre/post-fix states.

TCP State Machine Quick Reference

Key states and meanings:

LISTEN : Server waiting for client SYN

SYN_SENT : Client sent SYN, awaiting SYN+ACK

SYN_RECV : Server sent SYN+ACK, awaiting client ACK

ESTABLISHED : Connection fully established

FIN_WAIT_1 : Active closer sent FIN, awaiting ACK

FIN_WAIT_2 : Active closer received ACK, awaiting peer FIN

CLOSE_WAIT : Passive closer received FIN, awaiting application close

LAST_ACK : Passive closer sent FIN, awaiting final ACK

TIME_WAIT : Active closer waits 2MSL (default 60s) to ensure final ACK delivery

CLOSING : Simultaneous close (rare)

CLOSED : Fully closed

Critical numbers: Active closer enters TIME_WAIT for 2MSL (60s default). Passive closer stays in CLOSE_WAIT until application calls close (indefinite if not closed). Server SYN_RECV exceeding backlog triggers syncookies.

netstat Practical Usage

Installation & Version

# RHEL series
yum install -y net-tools

# Debian/Ubuntu
apt-get install -y net-tools

# Check version
netstat -V 2>&1 | head -1

Common Options

-a

: Show all sockets (listening + established) -n: Disable DNS reverse lookup, show numeric IP/port -t: TCP -u: UDP -x: Unix sockets -p: Show process info (requires root) -l: Only listening sockets -s: Protocol statistics -r: Routing table -i: Network interface statistics -c: Continuous refresh -e: Extended information

Classic Combinations

# All TCP connections (no DNS, fastest)
netstat -ant

# With process info (root)
netstat -antp

# Only listening
netstat -lntp

# Count by state
netstat -ant | awk '{print $NF}' | sort | uniq -c | sort -rn

# Who listens on port 80
netstat -lntp | grep ':80 '

# Clients connected to port 80
netstat -ant | grep ':80 '

# Routing table
netstat -rn

# Interface stats
netstat -i

Output Interpretation

Proto Recv-Q Send-Q Local Address          Foreign Address        State       PID/Program name
tcp        0      0 0.0.0.0:80             0.0.0.0:*              LISTEN      1234/nginx
tcp        0      0 10.0.0.5:80            10.0.1.100:54321       ESTABLISHED 1234/nginx

Recv-Q

: Receive queue bytes. In LISTEN, non-zero means accept queue full. Send-Q: Send queue bytes. Local Address: Local IP:port. Foreign Address: Peer IP:port. State: Connection state.

netstat Limitations

Very slow at 100k+ ESTABLISHED connections due to socket traversal and resolution attempts.

Fixed output format, not customizable.

Cannot show congestion window, retransmissions, or other internal TCP metrics.

These scenarios favor ss.

ss Practical Usage

Installation

yum install -y iproute   # RHEL
apt-get install -y iproute2  # Debian/Ubuntu

Common Options

-s

: Summary statistics -a: All sockets -l: Only listening -n: Disable DNS resolution -p: Process info -t: TCP -u: UDP -x: Unix sockets -4/-6: IPv4/IPv6 -i: Internal TCP info (rtt, cwnd, retrans) -o: Timer info -e: Extended socket info -m: Socket memory usage -z: Show process context with -p -K: Force close socket (use with caution!)

Classic Combinations

# Full summary (health checks)
ss -s

# TCP established count
ss -tan state established | wc -l

# Group count by state
ss -tan | awk 'NR>1 {print $1}' | sort | uniq -c | sort -rn

# Listening + process
ss -lntp

# Connections to port 22 (note colon)
ss -tan '( dport = :22 or sport = :22 )'

# All port 80 connections
ss -tan '( sport = :80 or dport = :80 )'

# Connections from specific IP
ss -tan dst 10.0.1.100

# Connections from subnet
ss -tan dst 10.0.0.0/16

# Socket memory usage
ss -tan -m | head

# TIME_WAIT count (connection pool issues)
ss -tan state time-wait | wc -l

# CLOSE_WAIT
ss -tan state close-wait | head

# SYN_RECV (suspected SYN Flood)
ss -tan state syn-recv | head

Internal TCP Info (Core)

# View RTT, cwnd, retrans for each ESTABLISHED connection
ss -tin

Example output:

ESTAB  0  0  10.0.0.5:22  10.0.0.100:54321  
    users:(("sshd",pid=1234,fd=4))
    cubic wscale:7,7 rto:204 rtt:0.5/0.75 ato:40 mss:1448 cwnd:10 send 2.5Mbps rcv_rtt:0.5 rcv_space:29200

cubic

: Congestion control algorithm (Cubic/Reno/BBR/H-TCP). wscale: Window scale factor. rto: Retransmission timeout. rtt: Round-trip time, average/variance. mss: Maximum segment size. cwnd: Congestion window. retrans: Retransmission count (position varies by version, may appear as retrans:0).

ss Timers

ss -to

Shows per-socket timers, e.g.:

ESTAB 0 0 10.0.0.5:22 10.0.0.100:54321 timer:(keepalive,28sec,0)

keepalive

: TCP keepalive timer. retrans: Retransmission timer. probe: Zero-window probe.

Emergency Connection Closure

# Close all connections on port 80 (CAUTION: drops live users)
ss -K 'dport = :80'

# Close all connections to specific IP
ss -K 'dst 10.0.1.100'

# Close connections in specific state
ss -K 'state time-wait'

Risks: -K forces closure equivalent to application close but without resource cleanup. In production, require change request, maintenance window, avoid peak hours. Always verify process ownership with ss -p first.

tcpdump Practical Usage

Installation

yum install -y tcpdump
apt-get install -y tcpdump

Basic Usage

# Capture all interfaces (production caution: data explosion)
tcpdump -i any

# Specific interface
 tcpdump -i eth0

# Exit after N packets
 tcpdump -i eth0 -c 100

# Save to pcap
 tcpdump -i eth0 -w /tmp/cap/cap.pcap

# Print to screen with verbose
 tcpdump -i eth0 -nn -vv

# Capture full packet (default 68 bytes truncates payload)
 tcpdump -i eth0 -s 0

BPF Filter Expressions

High-frequency operational filters:

# By host
 tcpdump -i any host 10.0.1.100

# By source/destination
 tcpdump -i any src 10.0.1.100
 tcpdump -i any dst 10.0.1.100

# By network
 tcpdump -i any net 10.0.0.0/16

# By port
 tcpdump -i any port 80
 tcpdump -i any src port 80
 tcpdump -i any dst port 80

# By protocol
 tcpdump -i any tcp
 tcpdump -i any udp
 tcpdump -i any icmp

# Combined conditions
 tcpdump -i any 'tcp and port 80 and host 10.0.1.100'

# By TCP flags
 tcpdump -i any 'tcp[tcpflags] & tcp-syn != 0'          # All SYN
 tcpdump -i any 'tcp[tcpflags] & tcp-syn != 0 and tcp[tcpflags] & tcp-ack == 0'  # SYN only
 tcpdump -i any 'tcp[tcpflags] & tcp-rst != 0'          # All RST
 tcpdump -i any 'tcp[tcpflags] & tcp-fin != 0'          # FIN

# By packet size
 tcpdump -i any 'greater 1000'   # >1000 bytes
 tcpdump -i any 'less 64'        # <64 bytes

# Payload content (risky, may false-match)
 tcpdump -i any -A 'tcp port 80 and (((ip[2:2] - ((ip[0]&0xf)<<2)) - ((tcp[12]&0xf0)>>2)) != 0)'

Advanced Filters

# Capture HTTP request lines
 tcpdump -i any -A -s 0 'tcp port 80 and (((ip[2:2] - ((ip[0]&0xf)<<2)) - ((tcp[12]&0xf0)>>2)) != 0)' 2>/dev/null | grep -E '^(GET|POST|HTTP)'

# Capture HTTP response codes
 tcpdump -i any -A -s 0 'tcp port 80 and (((ip[2:2] - ((ip[0]&0xf)<<2)) - ((tcp[12]&0xf0)>>2)) != 0)' 2>/dev/null | grep -E '^HTTP/1\.[01]'

# Capture MySQL queries
 tcpdump -i any -A 'tcp port 3306' 2>/dev/null | grep -E 'Query|SELECT'

# Capture Redis commands
 tcpdump -i any -A 'tcp port 6379' 2>/dev/null | grep -E '^\*'

Risks: -A ASCII output floods terminal at high packet rates; prefer writing pcap for Wireshark analysis. At >10k QPS, packet drops may occur; increase buffer with -B 4096 or capture single interface -i eth0.

Capture Scenarios

DNS Resolution

tcpdump -i any -nn -s 0 -w /tmp/cap/dns.pcap port 53

Analyze with tcpdump -nn -r /tmp/cap/dns.pcap or Wireshark.

HTTPS Handshake (No Decryption)

tcpdump -i any -nn -s 0 -w /tmp/cap/tls.pcap 'tcp port 443'

Decryption requires server SSLKEYLOGFILE (supported by openssl/nginx) imported into Wireshark.

SYN Flood

tcpdump -i any -nn -s 0 'tcp[tcpflags] & tcp-syn != 0 and tcp[tcpflags] & tcp-ack == 0' -c 1000 -w /tmp/cap/syn.pcap

Capture 1000 SYN packets, then analyze source IP distribution.

TCP Retransmissions

# tcpdump cannot directly label retransmissions, but shows duplicate SEQ
 tcpdump -i any -nn -vv -tttt 'tcp port 80' 2>&1 | grep -E 'retrans|duplicate' | head

Better: Open pcap in Wireshark which auto-labels "TCP Retransmission".

Output Format

10:23:45.123456 IP 10.0.1.100.54321 > 10.0.0.5.80: Flags [S], seq 123456, win 64240, options [mss 1460], length 0

10:23:45.123456

: Timestamp. IP: Protocol. 10.0.1.100.54321 > 10.0.0.5.80: Source > destination. Flags [S]: TCP flags. seq 123456: Sequence number. win 64240: Window size. length 0: Payload length.

Common flags: S =SYN, S. =SYN+ACK, . =ACK, F =FIN, R =RST, P =PSH, U =URG.

/proc Kernel-Level Connection View

When connections are extremely high and ss / netstat are slow, read kernel proc files directly.

Key Files

/proc/net/tcp

: TCP socket table /proc/net/tcp6: IPv6 TCP socket table /proc/net/udp: UDP socket table /proc/net/unix: Unix domain socket table /proc/net/sockstat: Socket counters /proc/net/netstat: Protocol layer statistics /proc/net/snmp: SNMP MIB statistics /proc/sys/net/ipv4/tcp_syncookies: Syncookies toggle (0/1) /proc/sys/net/ipv4/tcp_max_syn_backlog: Half-connection queue limit /proc/sys/net/core/somaxconn: Accept queue limit

Reading /proc/net/tcp

cat /proc/net/tcp

Output:

sl  local_address rem_address   st tx_queue rx_queue tr tm->when retrnsmt   uid  timeout inode
   0: 0100007F:0050 00000000:0000 0A 00000000:00000000 0:0 00000000     0      0 12345 1 ffff8800b6f0c000
   1: 0500000A:0050 6401000A:C8C5 01 00000000:00000000 0:0 00000000  1000      0 23456 1 ffff8800b6f0c100

sl

: Serial number. local_address: Local IP:port (hex, IP in network byte order little-endian). rem_address: Peer IP:port. st: State (hex).

State mapping:

01 = ESTABLISHED

02 = SYN_SENT

03 = SYN_RECV

04 = FIN_WAIT_1

05 = FIN_WAIT_2

06 = TIME_WAIT

07 = CLOSE

08 = CLOSE_WAIT

09 = LAST_ACK

0A = LISTEN

0B = CLOSING tx_queue / rx_queue: Send/receive queues.

Hex IP conversion example: 0100007F = 127.0.0.1 (read bytes reversed: 7F 00 00 01). 0050 = 80.

Conversion script:

cat /proc/net/tcp | awk 'NR>1 {
  split($2, l, ":")
  split($3, r, ":")
  printf "%s:%d -> %s:%d st=%s
",
    l[1], strtonum("0x" l[2]),
    r[1], strtonum("0x" r[2]),
    $4
}'

strtonum

is a gawk extension converting hex string to number.

Auxiliary Tools

lsof

# Sockets opened by process
lsof -i

# Specific port
lsof -i :80

# Specific process
lsof -p 1234 | grep -i socket

# Specific user
lsof -i -u mysql

iftop / nethogs / iptraf-ng

# Real-time traffic per connection
iftop -i eth0

# Real-time traffic per process
nethogs eth0

# Text-mode graphical stats
iptraf-ng

sar -n Series

# Interface RX/TX stats
sar -n DEV 1

# Interface errors
sar -n EDEV 1

# TCP statistics
sar -n TCP 1

# Socket statistics
sar -n SOCK 1

%ifutil

near 100% indicates saturated NIC.

nstat / ip -s

# TCP retransmission counters (kernel)
nstat -az | grep -E 'TcpRetrans|TcpOutRsts|TcpActiveOpens'

# Interface stats
ip -s link show eth0

TcpRetransSegs

continuously increasing = network loss or peer RTO.

conntrack

# Current conntrack table size
cat /proc/sys/net/netfilter/nf_conntrack_max
sysctl net.netfilter.nf_conntrack_count

# List current conntrack
conntrack -L

# Detail for specific connection
conntrack -L -d 10.0.1.100

If nf_conntrack_count approaches nf_conntrack_max, packets drop, manifesting as curl/ssh timeouts.

strace for connect/send/recv

# Trace process syscalls
strace -f -p 1234 -e trace=network,write,read

# Trace connect failures
strace -f -p 1234 -e trace=connect 2>&1 | grep -E 'sin_addr|EINPROGRESS|ETIMEDOUT|ECONNREFUSED'

ethtool

# Driver / speed
 ethtool eth0

# Detailed stats (driver-specific fields)
 ethtool -S eth0 | grep -E 'err|drop|carrier|crc'

# Queues and interrupts
 ethtool -l eth0
 ethtool -x eth0

Case Studies

Case 1: TIME_WAIT Surge, Connection Pool Exhaustion

Phenomenon

Java+Tomcat service alerts "cannot obtain connection", monitoring shows ~60k ESTABLISHED, ~40k TIME_WAIT, 5xx errors.

Initial Hypothesis

HTTP short connections, new connection per request.

Connection pool too small, frequent create/destroy.

Server-side keep-alive timeout closing connections.

Middleware (LB/NAT) changed session timeout.

Diagnostic Commands

# 1) TIME_WAIT count
ss -tan state time-wait | wc -l

# 2) Top peer IPs
ss -tan state time-wait | awk 'NR>1 {print $5}' | cut -d: -f1 | sort | uniq -c | sort -rn | head

# 3) Local port distribution
ss -tan state time-wait | awk 'NR>1 {print $4}' | cut -d: -f2 | sort | uniq -c | sort -rn | head

# 4) Process ownership
ss -tanp state time-wait | head

# 5) Kernel global stats
nstat -az | grep -E 'TcpActiveOpens|TcpPassiveOpens|TcpTW'

Key Indicators

TcpTW

: Cumulative TIME_WAIT entries, spike indicates anomaly. ss -tan state time-wait | wc -l: Instantaneous TIME_WAIT count.

Local port concentration → same service (e.g., reverse proxy → upstream).

Foreign address concentration → same peer (e.g., calling an API gateway).

Root Cause

Local ports clustered around 8080, foreign addresses clustered at LB ingress IP. Conclusion: Upstream LB actively closes after keep-alive timeout, causing downstream TIME_WAIT surge.

Evidence: nstat -az | grep TcpTW growth matches LB active-close rhythm. Packet capture on LB confirms FIN originates from LB.

Fix

Short-term:

# Enable TIME_WAIT reuse (Linux default allows, but verify)
sysctl -w net.ipv4.tcp_tw_reuse=1

# Reduce TIME_WAIT duration (default 60s)
sysctl -w net.ipv4.tcp_fin_timeout=15

Note: tcp_tw_recycle causes false kills in NAT, removed in Linux 4.12+, never enable.

Long-term:

Client: Use persistent connections + connection pool.

Server: Increase keepalive_timeout (Nginx).

Evaluate if keep-alive needed; consider long-polling or SSE for some APIs.

Verification

# After 5 minutes
ss -tan state time-wait | wc -l   # Should drop significantly
ss -s                              # Summary TIME_WAIT back to normal

# Business verification
curl -sSI https://api.example.com/health

Retrospective

Persist sysctl changes in /etc/sysctl.d/ (lost on reboot).

TIME_WAIT is TCP reliability guarantee, not a bug; goal is control, not elimination.

Alert when TIME_WAIT/ESTABLISHED ratio exceeds 0.5.

Case 2: SYN_RECV Buildup, Suspected SYN Flood

Phenomenon

Reverse proxy SYN_RECV grows from 0 to 8000+, new connections fail, existing latency rises.

Initial Hypothesis

Client intentionally withholds ACK (SYN Flood).

Client RST/disconnect/firewall drops ACK.

NAT state evicted.

Backend application slow, filling handshake queue.

Diagnostic Commands

# 1) SYN_RECV count
ss -tan state syn-recv | wc -l

# 2) Source distribution
ss -tan state syn-recv | awk 'NR>1 {print $5}' | cut -d: -f1 | sort | uniq -c | sort -rn | head

# 3) Packet capture confirmation
 tcpdump -i eth0 -nn -c 1000 'tcp[tcpflags] & tcp-syn != 0 and tcp[tcpflags] & tcp-ack == 0' -w /tmp/cap/syn.pcap

# 4) Check syncookies
 cat /proc/sys/net/ipv4/tcp_syncookies

# 5) Half-connection queue limits
 cat /proc/sys/net/ipv4/tcp_max_syn_backlog
 cat /proc/sys/net/core/somaxconn

Key Indicators

SYN_RECV > backlog threshold triggers syncookies. syncookies=1 means kernel using syncookies (resource cost but mitigates attack).

Small backlog increases drops; raise somaxconn and tcp_max_syn_backlog.

Root Cause

Capture shows SYN from 200+ botnet IPs, each ~30 SYN/sec, random distribution. Confirmed SYN Flood.

Fix

Emergency:

# Enable syncookies
sysctl -w net.ipv4.tcp_syncookies=1

# Increase half-connection queue
sysctl -w net.ipv4.tcp_max_syn_backlog=4096
sysctl -w net.core.somaxconn=4096

# Temporary iptables rate-limit per source IP SYN
iptables -I INPUT -p tcp --syn -m limit --limit 10/s --limit-burst 20 -j ACCEPT
iptables -A INPUT -p tcp --syn -j DROP

Long-term:

Cloud WAF / Anti-DDoS IP.

Nginx increase listen backlog.

Client IP blocklist aligned with threat intel.

Verification

# After 5 minutes
ss -tan state syn-recv | wc -l   # Should normalize
nstat -az | grep -E 'TcpExtTCPBacklogDrop|TcpSyncookiesSent'

Retrospective

SYN_RECV surge not always SYN Flood; combine source IP distribution and business impact.

Syncookies is last resort, degrades performance but better than outage.

iptables rate-limit is temporary; can be bypassed by ACK/UDP floods; long-term requires WAF.

Case 3: CLOSE_WAIT Persistent Growth

Phenomenon

Server process CLOSE_WAIT grows continuously, exceeds 50k after hours, new connections fail.

Initial Hypothesis

CLOSE_WAIT = passive closer received FIN, waiting for application close(). Persistent growth means application not closing socket. Common causes:

Code logic: Exception path fails to close socket.

Connection leak: close() not executed on error path.

Third-party library: HTTP client / DB pool not releasing properly.

JVM GC pause: Connection object GC'd but not closed.

Diagnostic Commands

# 1) CLOSE_WAIT count
ss -tan state close-wait | wc -l

# 2) Local port distribution (which service)
ss -tan state close-wait | awk 'NR>1 {print $4}' | cut -d: -f2 | sort | uniq -c | sort -rn | head

# 3) Process ownership
ss -tanp state close-wait | head

# 4) File descriptor count
ls /proc/$PID/fd | wc -l

# 5) Java: jstack for blocked threads
jstack $PID | grep -A 30 BLOCKED

Key Indicators

CLOSE_WAIT trend (increasing = abnormal).

Per-process FD count ( ls /proc/$PID/fd | wc -l).

Long-blocked threads in jstack.

DB pool active connections (Druid/HikariCP metrics).

Root Cause

Java application, jstack shows many threads stuck in HttpClient.readResponse, confirming HTTP client not releasing connection on exception path.

Fix

# Temporary relief: restart service (drain traffic first!)
systemctl restart myservice

# Code fix: ensure finally closes, or use try-with-resources

Correct pattern:

try (CloseableHttpResponse resp = httpClient.execute(request)) {
    // process
}

Verification

ss -tan state close-wait | wc -l   # Zero after restart
ss -s                              # Summary close-wait = 0

Retrospective

CLOSE_WAIT root cause is application layer, not network.

Add monitoring alert on state close-wait count.

Set ulimit -n at service startup to prevent FD exhaustion.

Case 4: Cross-Host Latency High, NIC Soft Interrupt Imbalance

Phenomenon

Same datacenter, ping avg 1ms, business call P99 200ms+.

Diagnostic Commands

# 1) NIC stats
sar -n DEV 1 5
# Focus on %ifutil, rx/s, tx/s

# 2) Packet drops
netstat -i
ip -s link show eth0 | grep -E 'drop|err|fifo'

# 3) Soft interrupt distribution
cat /proc/interrupts | grep eth0
mpstat -P ALL 1

# 4) RPS/RFS enabled?
cat /proc/sys/net/core/rps_sock_flow_entries
ls /sys/class/net/eth0/queues/

# 5) tcpdump for retransmissions
 tcpdump -i eth0 -nn -c 1000 'tcp port 80' -w /tmp/cap/cap.pcap

Key Indicators

%ifutil

near 100%. drop counter rising fast.

Single CPU softirq >80%.

tcpdump shows many TCP retransmissions.

Root Cause

Single-queue NIC on multi-core system concentrates all traffic on one core, causing softirq bottleneck. Enabling RPS (Receive Packet Steering) alleviates.

Fix

# Temporary enable RPS
echo ffff > /sys/class/net/eth0/queues/rx-0/rps_cpus

# Persist (add to /etc/rc.local or udev rule)

Or use multi-queue NIC + tuning:

# Multi-queue tuning
ethtool -L eth0 combined 8

Verification

sar -n DEV 1   # %ifutil drops
mpstat -P ALL 1  # Softirq distributed across cores

Case 5: HTTPS Handshake Failure

Phenomenon

Caller reports intermittent HTTPS failures: SSL handshake failed or connection reset.

Diagnostic Commands

# 1) Capture TLS handshake
 tcpdump -i eth0 -nn -s 0 -w /tmp/cap/tls.pcap 'tcp port 443'

# 2) Active openssl test
 openssl s_client -connect api.example.com:443 -servername api.example.com -msg -debug 2>&1 | head -50

# 3) Server certificate info
 openssl x509 -in /etc/nginx/ssl/server.crt -text -noout

# 4) curl verbose handshake
 curl -v --tlsv1.2 https://api.example.com 2>&1 | grep -E 'TLS|SSL|cipher'

Key Indicators

Complete ClientHello/ServerHello exchange?

Alert messages (21-70 different levels)?

RST position: during handshake usually certificate/protocol mismatch.

Root Cause

Capture shows ServerHello then immediate RST. openssl s_client reveals: SSL alert number 42 Alert 42 = bad_certificate. Server rejects client certificate. Investigation finds client CA bundle missing intermediate certificate.

Fix

Client: Complete CA bundle.

Server: Enable SSL Stapling ( ssl_stapling on;).

Verification

openssl s_client -connect api.example.com:443 -CAfile /etc/pki/tls/certs/ca-bundle.crt
# Verify return code: 0 (ok)

Kernel Parameter Tuning Cheatsheet

Common Parameters

net.core.somaxconn

: Accept queue limit (default 128 old kernels) → 1024-4096 net.ipv4.tcp_max_syn_backlog: Half-connection queue limit (default 128) → 4096 net.ipv4.tcp_synack_retries: SYN+ACK retry count (default 5) → 2-3 net.ipv4.tcp_syncookies: Syncookies toggle (default 1) → 1 (production) net.ipv4.tcp_tw_reuse: TIME_WAIT reuse (default 0) → 1 (NAT clients) net.ipv4.tcp_fin_timeout: FIN_WAIT_2 timeout (default 60) → 15-30 net.ipv4.tcp_keepalive_time: Keepalive start time (default 7200) → 600 net.ipv4.tcp_keepalive_intvl: Keepalive interval (default 75) → 30 net.ipv4.tcp_keepalive_probes: Keepalive probe count (default 9) → 3 net.ipv4.ip_local_port_range: Local port range (default 32768 60999) → 1024 65535 net.ipv4.tcp_max_tw_buckets: TIME_WAIT limit (default 262144) → 200000 net.ipv4.tcp_slow_start_after_idle: Slow start after idle (default 1) → 0 (disable) net.ipv4.tcp_fastopen: TFO toggle (default 0) → 1 (on demand) net.ipv4.tcp_mtu_probing: MTU probing (default 0) → 1 (on demand) net.core.rmem_max / wmem_max: Socket max buffer (default 212992) → 16777216 net.ipv4.tcp_rmem / tcp_wmem: TCP auto-tuning buffers (default 4096 87380 6291456) → 4096 87380 16777216

Modification Procedure

# 1. Backup
cp -a /etc/sysctl.d/99-sysctl.conf /tmp/sysctl.conf.bak

# 2. Edit
cat >> /etc/sysctl.d/99-network-tuning.conf <<'EOF'
net.core.somaxconn = 4096
net.ipv4.tcp_max_syn_backlog = 4096
net.ipv4.tcp_synack_retries = 2
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_fin_timeout = 15
EOF

# 3. Apply
sysctl -p /etc/sysctl.d/99-network-tuning.conf

# 4. Verify
sysctl net.core.somaxconn

Risks: Confirm business doesn't rely on defaults. Canary deploy: test cluster → production canary → full rollout. Avoid production peak hours.

Persistence & Rollback

# Persist
echo "net.ipv4.tcp_tw_reuse = 1" >> /etc/sysctl.d/99-network-tuning.conf
sysctl -p

# Rollback
mv /etc/sysctl.d/99-network-tuning.conf /etc/sysctl.d/99-network-tuning.conf.disabled
sysctl -p

Packet Analysis Patterns

TCP Three-Way Handshake

10:00:00.000 IP 10.0.0.100.54321 > 10.0.0.5.80: Flags [S], seq 100, length 0
10:00:00.001 IP 10.0.0.5.80 > 10.0.0.100.54321: Flags [S.], seq 200, ack 101, length 0
10:00:00.001 IP 10.0.0.100.54321 > 10.0.0.5.80: Flags [.], ack 201, length 0

SYN → SYN+ACK → ACK = connection established.

TCP Four-Way Teardown

10:00:05.000 IP client > server: Flags [F.], seq 1000, length 0
10:00:05.001 IP server > client: Flags [.], ack 1001, length 0
10:00:10.000 IP server > client: Flags [F.], seq 2000, length 0
10:00:10.001 IP client > server: Flags [.], ack 2001, length 0

TCP Retransmission

10:00:00.000 IP server > client: Flags [.], seq 1000, ack 500, length 1000
10:00:00.500 IP server > client: Flags [.], seq 1000, ack 500, length 1000  # retrans
10:00:01.500 IP server > client: Flags [.], seq 1000, ack 500, length 1000  # retrans

Same SEQ+LEN repeated = TCP retransmission.

Client Receives RST

10:00:00.000 IP client > server: Flags [S], seq 100
10:00:00.001 IP server > client: Flags [S.], seq 200, ack 101
10:00:00.002 IP server > client: Flags [R], seq 201, length 0

Server RST after SYN+ACK. Causes: SYN cookie validation failure, application reject, kernel security policy.

Containerized Network Troubleshooting

Pod Packet Capture

Containers lack tcpdump by default. Options:

# Method 1: Manual install (not recommended, modifies image)
kubectl exec -it mypod -- apt-get update && apt-get install -y tcpdump

# Method 2: nsenter (host tools)
PID=$(kubectl get pod mypod -o jsonpath='{.status.containerStatuses[0].containerID}' | sed 's/.*///')
nsenter -n -t $PID tcpdump -i any -nn 'port 80'

CNI Network Issues

Cilium/Calico/Flannel characteristics:

Calico: BGP or VXLAN, issues often at routing layer.

Cilium: eBPF data plane, issues often in bpf map state.

Flannel: Simple VXLAN, moderate performance but stable.

Troubleshooting steps:

# Pod IP vs Node IP
kubectl get pod -o wide

# CNI logs (Calico example)
kubectl logs -n kube-system -l k8s-app=calico-node

# iptables rules (kube-proxy generated)
iptables-save | head -50

Service/Ingress Issues

# Service endpoints
kubectl get endpoints myservice

# Endpoint details
kubectl describe svc myservice

# Ingress status
kubectl describe ingress myingress

Common problems: Endpoints: <none>: Selector mismatch, Pod not Ready. 503: Ingress controller forward failure, usually upstream unreachable.

Production Packet Capture Guidelines

Disk Space : Capturing 10GB traffic needs 20-30GB spare.

Capture Rate : At 1Gbps line rate, tcpdump may drop packets (limited kernel ring buffer). Use tcpdump -B 4096 -i eth0 ... to increase buffer.

Business Impact : Capture consumes 5-15% CPU; avoid peak hours.

Compliance : Captures contain user data (HTTP plaintext/API calls); follow data governance.

Data Security : pcap files hold sensitive data; encrypt storage, purge regularly.

Change Process : File change request, schedule off-peak, backup data, archive analysis.

Post-Mortem Archive : Store captures + analysis in knowledge base for team reuse.

iptables/nftables & Connection Tracking

View conntrack

conntrack -L | head
conntrack -S | head

Check nf_conntrack Saturation

sysctl net.netfilter.nf_conntrack_count
sysctl net.netfilter.nf_conntrack_max

count

near max causes drops.

Emergency Increase

sysctl -w net.netfilter.nf_conntrack_max=524288
sysctl -w net.netfilter.nf_conntrack_tcp_timeout_established=3600

Key iptables Rules

# View SYN rate limits
iptables -L -n -v | grep -E 'SYN|conn'

# Rule hit counts
iptables -L -n -v

pkts

and bytes growing = rule active; zero = rule not matching.

Monitoring & Alerting Recommendations

Essential Metrics

node_netstat_Tcp_CurrEstab

: Current ESTABLISHED — per business node_netstat_Tcp_ActiveOpens: Cumulative active opens — spike alert node_netstat_Tcp_PassiveOpens: Cumulative passive opens — spike alert node_netstat_Tcp_RetransSegs: Cumulative retransmitted segments — growth rate alert node_netstat_Tcp_CurrEstab: Current ESTABLISHED — near ulimit alert node_netstat_Tcp_InErrs: Input packet errors — growth alert node_netstat_Tcp_OutRsts: Output RST packets — spike alert node_conntrack_count: Conntrack entries — near max alert node_sockstat_TCP_alloc: Allocated TCP sockets — spike alert

Prometheus Scrape Config

scrape_configs:
- job_name: 'node'
  static_configs:
  - targets: ['10.0.0.5:9100']

node_exporter includes netstat metrics.

Custom Alert Rules

groups:
- name: network
  rules:
  - alert: HighTcpRetrans
    expr: rate(node_netstat_Tcp_RetransSegs[5m]) > 100
    for: 5m
    labels:
      severity: warning
    annotations:
      summary: "TCP retransmission rate high"
      description: "{{ $labels.instance }} retrans rate {{ $value }}/s"

FAQ

Q1: netstat vs ss?

Prefer ss. netstat unusable above 100k connections. ss reads kernel hash table directly, faster, script-friendly output.

Q2: Why tcpdump captures nothing?

Promiscuous mode not enabled (usually not needed).

BPF expression error: start simple, add constraints gradually.

Wrong capture point: traffic not traversing that interface. Example: two pods on same node communicating — physical NIC sees nothing.

Q3: How to view TCP internals (RTT, cwnd)?

ss -tin

Or use tcpprobe / perf kernel tracing.

Q4: How to decrypt HTTPS content?

Requires server SSLKEYLOGFILE + Wireshark decryption. Alternatively, use openssl / curl active handshake tests to bypass production capture.

Q5: Is TIME_WAIT harmful?

TIME_WAIT is TCP reliability guarantee preventing final ACK loss. Short connections → many TIME_WAIT. Optimization: reduce short connections (persistent connections + pool), not eliminate TIME_WAIT.

Q6: Emergency CLOSE_WAIT mitigation?

# Find process, restart (drain traffic first!)
kill -TERM $PID

Root fix in code (finally/try-with-resources).

Q7: Why many RST?

Server port not listening.

Kernel security policy (SYN Cookie/iptables reject).

Application actively closes incomplete sockets.

Mass keepalive probe failures.

Q8: conntrack full emergency?

sysctl -w net.netfilter.nf_conntrack_max=1048576

But first diagnose why full: scanner attacks, DNS cache failure, excessive UDP traffic.

Command Cheatsheet

# Connection overview
ss -s

# Count by state
ss -tan | awk 'NR>1 {print $1}' | sort | uniq -c

# All connections in specific state
ss -tan state time-wait
ss -tan state close-wait
ss -tan state syn-recv

# By IP/subnet
ss -tan dst 10.0.1.100
ss -tan dst 10.0.0.0/16

# Emergency close
ss -K 'dport = :80'
ss -K 'dst 10.0.1.100'

# Internal TCP info
ss -tin

# Packet capture
 tcpdump -i eth0 -nn -s 0 -w /tmp/cap/cap.pcap port 80
 tcpdump -i any 'tcp and port 80 and host 10.0.1.100'

# BPF filters
 tcpdump 'tcp[tcpflags] & tcp-syn != 0'          # All SYN
 tcpdump 'greater 1000'                           # >1000 bytes
 tcpdump 'less 64'                                # <64 bytes

# NIC / softirq
sar -n DEV 1
mpstat -P ALL 1
cat /proc/interrupts | grep eth0

# conntrack
sysctl net.netfilter.nf_conntrack_count
sysctl net.netfilter.nf_conntrack_max
conntrack -L | head

# Kernel parameters
sysctl -a | grep net.ipv4.tcp
sysctl -p /etc/sysctl.d/99-network-tuning.conf

# Kernel TCP stats
nstat -az | grep -E 'TcpRetrans|TcpActiveOpens|TcpPassiveOpens|TcpOutRsts'

Summary & Best Practices

ss first, then tcpdump : ss is "macro checkup", tcpdump is "biopsy". If ss reveals it, don't capture.

Capture small then large : Start with -c 1000 for 1000 packets, then decide on full capture.

Internalize TCP state machine : Know it cold; seeing CLOSE_WAIT immediately signals application issue.

Always use BPF : Narrow scope with BPF or data won't fit on disk.

sysctl changes via canary : Test cluster → production canary → full rollout, keep rollback commands.

Production capture requires change request : Document, avoid peak, backup, archive results.

Cross-validate multiple metrics : ESTABLISHED up + TIME_WAIT down + nstat TcpTW rising — all three together give confidence.

Adopt BBR : For high-bandwidth long-fat networks, BBR significantly outperforms CUBIC.

Monitor conntrack religiously : Many "mysterious timeouts" are conntrack exhaustion.

Maintain team knowledge base : Archive every investigation's ss / tcpdump output, parameters, reasoning for reuse.

Script daily health checks : Write healthcheck.sh running daily, logging anomalous metrics.

Don't worship single metrics : High ESTABLISHED isn't necessarily bad; high TIME_WAIT isn't necessarily bad. Combine with business P99, error rates.

The highest level of network troubleshooting isn't using fancy tools — it's "getting the most direct evidence with the fewest commands". Every command combination in this article serves that goal.