How One tcpdump Command Ended a 3‑Day Network Outage (Full Linux Network Toolkit)

Basic Network Commands: The Ops Swiss‑Army Knife

When a network issue appears, the first step is to use simple commands to narrow the scope. Commonly used commands include:

1. ping – Connectivity testing

# Send 4 packets then stop
ping -c 4 baidu.com
# Send a packet every 0.5 s for denser testing
ping -i 0.5 192.168.1.1
# Specify packet size to test MTU
ping -s 1472 -M do 8.8.8.8

Note: A failed ping does not always mean the network is down; many servers block ICMP, so try telnet or curl to the port.

2. traceroute – Trace the packet path

# Basic usage
traceroute google.com
# Use TCP SYN packets (easier through firewalls)
traceroute -T -p 443 api.example.com
# Skip DNS resolution for speed
traceroute -n 8.8.8.8

3. ip (preferred over ifconfig)

# Show all interfaces
ip addr show
# Show only eth0
ip addr show eth0
# Show interface state (up/down)
ip link show
# Show routing table
ip route show

4. ss / netstat – Port and connection status

# List all listening TCP/UDP ports
ss -tuln
# Show all TCP connections with process names
ss -tulp
# Show connection statistics
ss -s
# Show connections on port 80
ss -tuln | grep :80

tcpdump: The Linux Packet‑Capture Workhorse

If basic commands cannot locate the problem, packet capture is required. tcpdump runs on the server without a GUI.

1. Basic capture commands

# Capture all interfaces and write to file
tcpdump -i any -w capture.pcap
# Capture only eth0, stop after 100 packets
tcpdump -i eth0 -c 100
# Disable name resolution for speed
tcpdump -nn
# Verbose output with protocol details
tcpdump -vv

2. Filter expressions – Precise packet selection

# By port
tcpdump port 80
tcpdump port 443
tcpdump src port 12345
tcpdump dst port 80
# By IP
tcpdump host 192.168.1.100
tcpdump src host 10.0.0.5
tcpdump dst host 172.16.0.10
# By protocol
tcpdump tcp
tcpdump udp
tcpdump icmp
# Combined conditions
tcpdump host 192.168.1.100 and port 443
tcpdump port 80 or port 443
tcpdump not port 22
# Fine‑grained filters
tcpdump "tcp[tcpflags] & tcp-syn != 0"
tcpdump "tcp[tcpflags] & tcp-rst != 0"

Practical tip: For continuous capture, use tcpdump -i eth0 -nn -w dump.pcap -C 100 -W 10 to rotate 100 MB files, keeping at most ten.

Wireshark: Graphical Deep‑Dive Analyzer

tcpdump captures packets; Wireshark decodes them. Its powerful filter language and protocol dissection make it ideal for complex issues.

1. Common filter syntax

# IP filters
ip.addr == 192.168.1.100
ip.src == 10.0.0.5
ip.dst == 172.16.0.10
# Port filters
tcp.port == 443
tcp.srcport == 12345
tcp.dstport == 80
# Protocol filters
http
tls
dns
icmp
# TCP flag filters
tcp.flags.syn == 1
tcp.flags.reset == 1
tcp.flags.fin == 1
# Combined conditions
ip.addr == 192.168.1.100 and tcp.port == 443
http.request.method == "POST"
http.response.code == 404
# Quick analysis filters
tcp.analysis.retransmission
tcp.analysis.duplicate_ack
tcp.analysis.zero_window

2. Follow TCP Stream

Right‑click a packet → Follow → TCP Stream to view the full conversation, including HTTP requests/responses or TLS handshakes.

3. IO Graph

Statistics → IO Graph visualizes traffic spikes and retransmission periods.

Protocol‑Specific Analysis Tips

1. TCP Handshake / Teardown

Normal three‑way handshake: SYN → SYN‑ACK → ACK. Missing SYN‑ACK indicates a closed port or firewall block; many SYN packets may signal a SYN‑flood attack.

2. HTTP

Plain‑text; use “Follow HTTP Stream” to see request/response headers and bodies. Check status codes, Time‑to‑First‑Byte, and Content‑Length consistency.

3. DNS

Filter UDP port 53. Verify queries are sent, responses received, response time (< 100 ms), and correct IP addresses.

4. TLS/HTTPS Handshake

Handshake is visible in clear text. Look for Certificate messages, Alert messages, and handshake RTT; TLS 1.3 saves one RTT compared to TLS 1.2.

Real‑World Cases

Case 1: curl hangs while telnet succeeds

Symptom: curl blocks, telnet shows the port is open.

Analysis: tcpdump shows a completed three‑way handshake, HTTP request sent, then the server returns an RST.

Root cause: Nginx’s client_max_body_size limit rejected the request body, causing an immediate RST.

Fix: Increase client_max_body_size or compress the request payload.

Case 2: Intermittent timeout

Symptom: Same API sometimes returns in 100 ms, other times takes seconds or times out.

Analysis: Filtering for tcp.analysis.retransmission reveals many retransmissions concentrated in certain periods; IO Graph confirms spikes.

Root cause: Mismatched duplex settings between server NIC and switch port (half‑duplex vs full‑duplex) causing collisions and packet loss.

Fix: Force both ends to 1000 Mbps full‑duplex; the issue disappears.

Case 3: HTTPS handshake failure – “connection reset”

Symptom: Chrome shows ERR_CONNECTION_RESET; curl returns “Connection reset by peer”.

Analysis: Capture shows Client Hello, Server Hello, Certificate, then an immediate RST. Wireshark displays a Fatal Alert “Handshake Failure”.

Root cause: Incomplete certificate chain – only the server certificate was configured, missing intermediate certificates.

Fix: Concatenate server, intermediate, and root certificates in the correct order; handshake succeeds.

General Troubleshooting Workflow

Confirm it’s not a local issue : check interface IP, ping gateway, test DNS with nslookup or dig.

Verify network‑layer connectivity : ping target IP (beware of ICMP blocks), telnet or nc to the port, run traceroute to locate the problematic hop.

Check application layer : curl the endpoint, inspect HTTP status, review server logs, look for firewall packet drops.

If still unresolved, capture packets : capture on both client and server, apply filters, focus on SYN/ACK, RST, and retransmissions.

Experience tip: 90 % of network problems stem from firewalls, routing, or DNS. Prioritizing these three areas yields the fastest resolution.

Conclusion

Network troubleshooting is a core Ops skill. Mastering the basic commands narrows the problem space quickly, while tcpdump and Wireshark provide deep insight to identify root causes. The workflow and examples above cover the majority of everyday scenarios.