Using tcpdump to Diagnose Client‑Server Communication Failures

1. Define the problem

Record failure time window, source and destination IPs, ports, domain, request ID, reproduction steps, and capture location. Conclusions should describe observable facts (e.g., “server NIC did not see the SYN for the five‑tuple”) rather than attributing loss without joint evidence.

2. Verify tcpdump version and interfaces

#!/usr/bin/env bash
set -euo pipefail
command -v tcpdump
tcpdump --version
tcpdump --help | sed -n '1,80p'
ip -brief link
ip -brief address

Install tcpdump from the distribution repository and run it with root or CAP_NET_RAW/CAP_NET_ADMIN privileges.

3. Identify the actual egress interface

In multi‑NIC, policy routing, VPN, or container overlay environments, the default route may not carry the target traffic. Resolve the destination and query the kernel for the selected path.

TARGET="<target domain or server IP>"
getent ahosts "$TARGET"
ip route get <server IP>
ip rule show

If NAT, layer‑4 proxy, or load balancer is involved, the source address seen by the server may differ from the client’s IP; confirm the actual peer using gateway logs.

4. Map listening sockets to processes

sudo ss -Htanp '( sport = :<port> or dport = :<port> )'
sudo ss -Hlunp '( sport = :<port> or dport = :<port> )'

LISTEN state only proves a process is bound; address, routing, firewall, ACL, and service health may still cause external failures.

5. Create a controlled capture directory

#!/usr/bin/env bash
set -euo pipefail
CAPTURE_DIR="<capture directory>"
MIN_FREE_MIB="2048"
install -d -m 0700 "$CAPTURE_DIR"
FREE_MIB="$(df -Pm "$CAPTURE_DIR" | awk 'NR==2 {print $4}')"
if (( FREE_MIB < MIN_FREE_MIB )); then
  echo "insufficient free space: $FREE_MIB MiB" >&2
  exit 1
fi
date -Is > "$CAPTURE_DIR/capture-started-at.txt"

Agree on retention period and access scope; when handing PCAPs to external parties, redact or limit the capture.

6. Use narrow BPF filters to confirm packet existence

sudo tcpdump -i <interface> -nn -vv -c 100 'host <client IP> and host <server IP> and tcp port <port>'

The filter runs in the kernel and discards unrelated traffic. Verify the filter works with a small packet count before capturing full traces.

7. Capture only the TCP handshake

sudo tcpdump -i <interface> -nn -s 128 -vvv -w <capture dir>/tcp-handshake-$(date +%Y%m%d-%H%M%S).pcap 'host <client IP> and host <server IP> and tcp port <port>'

Only the header is needed for handshake analysis; full payload increases disk usage and data exposure.

8. Observe SYN, SYN‑ACK, RST and FIN

sudo tcpdump -i <interface> -nn -vv 'tcp port <port> and (tcp[tcpflags] & (tcp-syn|tcp-ack) = tcp-syn or tcp[tcpflags] & (tcp-syn|tcp-ack) = (tcp-syn|tcp-ack) or tcp[tcpflags] & tcp-rst != 0 or tcp[tcpflags] & tcp-fin != 0)'

Check the sequence: client SYN, server SYN receipt, server SYN‑ACK, client ACK, then any early RST or duplicate SYN. Compare timestamps and retransmission intervals on both ends.

sudo tcpdump -i <interface> -nn -vv 'host <client IP> and host <server IP> and tcp port <port> and tcp[tcpflags] & tcp-rst != 0'

RST may originate from the application, kernel, load balancer, or security device; correlate MAC/IP, TTL, interface, and logs before attributing cause.

9. Read timestamps and sequence numbers offline

tcpdump -nn -tttt -r <pcap file>.pcap 'host <client IP> and host <server IP> and tcp port <port>'

Use -S for absolute sequence numbers only when necessary, as they increase correlation risk.

10. Capture DNS, ICMP and layer‑2 neighbors

sudo tcpdump -i <interface> -nn -vv '(udp port 53 or tcp port 53) and host <DNS server IP>'

sudo tcpdump -i <interface> -nn -vv '(icmp or icmp6) and host <server IP>'

sudo tcpdump -i <interface> -enn -vv 'arp or icmp6'
ip neigh show dev <interface>

Frequent ARP requests without replies may indicate VLAN issues, duplicate IPs, or switch problems; verify MAC tables and port status.

11. Capture TLS, HTTP and UDP traffic

sudo tcpdump -i <interface> -nn -s 512 -vvv -w <capture dir>/tls-$(date +%Y%m%d-%H%M%S).pcap 'host <server IP> and tcp port 443'

sudo tcpdump -i <interface> -nn -s 512 -A 'host <client IP> and host <server IP> and tcp port 80'

sudo tcpdump -i <interface> -nn -vv -s 256 'udp and host <server IP> and port <port>'

TLS captures cannot reveal application payload; focus on handshake packets. For UDP, examine request/response pairs, intervals, lengths, and ICMP errors rather than applying TCP logic.

12. Use ring buffer capture when the issue is not immediately reproducible

#!/usr/bin/env bash
set -euo pipefail
INTERFACE="<interface>"
CAPTURE_DIR="<capture directory>"
FILTER='host <client IP> and host <server IP> and tcp port <port>'
install -d -m 0700 "$CAPTURE_DIR"
exec sudo tcpdump -i "$INTERFACE" -nn -s 256 -C 100 -W 12 -w "$CAPTURE_DIR/incident.pcap" "$FILTER"

Rotate files to avoid disk exhaustion; monitor kernel drop counters and adjust filters or buffer sizes if drops occur.

13. Capture inside containers and Kubernetes

CONTAINER_ID="<container ID or name>"
PID="$(docker inspect -f '{{.State.Pid}}' "$CONTAINER_ID")"
echo "container pid=$PID"
sudo nsenter -t "$PID" -n ip -brief address
sudo nsenter -t "$PID" -n ip route

sudo nsenter -t <container PID> -n tcpdump -i any -nn -s 256 'host <server IP> and tcp port <port>'

kubectl get pod <Pod name> -n <namespace> -o wide
kubectl get pod <Pod name> -n <namespace> -o jsonpath='{range .status.containerStatuses[*]}{.name}{"\t"}{.containerID}{"
"}{end}'

kubectl debug -n <namespace> -it pod/<Pod name> --target=<container name> --image=<trusted debug image> -- tcpdump -i any -nn -s 256 'tcp port <port>'

Enter the pod’s network namespace from the host for controlled capture; ensure RBAC, image provenance, audit and cleanup policies.

14. Disable offload features when they bias observations

sudo ethtool -k <interface> | grep -E 'tcp-segmentation-offload|generic-segmentation-offload|generic-receive-offload|rx-checksumming|tx-checksumming'

sudo ethtool -K <interface> gro off gso off tso off
sudo ethtool -k <interface> | grep -E 'generic-receive-offload|generic-segmentation-offload|tcp-segmentation-offload'

Turning off offload may affect throughput and CPU; perform only in a maintenance window, record original settings, and roll back after verification.

15. Preserve evidence and handoff conclusions

#!/usr/bin/env bash
set -euo pipefail
PCAP="<capture file>.pcap"
test -s "$PCAP"
chmod 0600 "$PCAP"
tcpdump -r "$PCAP" -c 1 > /dev/null
sha256sum "$PCAP" > "$PCAP.sha256"
ls -lh "$PCAP" "$PCAP.sha256"

Record capture point, interface, filter, start/end times, clock source, reproduction steps, request ID, PCAP hash, handshake comparison, and concurrent system logs. If tcpdump reports non‑zero “packets dropped by kernel,” the capture is incomplete and must be redone with tighter filters or a better capture point.