How to Diagnose Network Packet Loss: Practical Steps from ping to tcpdump

1. Phenomenon and Information Collection

Record start/end time, timezone, source/destination, protocol port, error type, and recent changes. Distinguish observations: ping loss only indicates missing ICMP Echo Replies, not necessarily TCP/HTTP loss.

Slow connection establishment or SYN timeout points to path, ACL, NAT, listener or conntrack issues.

Retransmissions in an established connection require checking both forward data and reverse ACKs.

HTTP timeout may stem from DNS, connection pool, application threads, GC or database, not only the network.

Run read‑only collection commands on both hosts and save the output with the failure window:

date -Is
hostnamectl --static
ip -br address
ip route get 10.30.2.18
ip -s link show dev eth0
ethtool eth0
ethtool -S eth0
ss -s
nstat -az

Key points: ip route get shows the kernel‑selected egress interface, source address and next hop; ethtool -S fields are driver‑specific and cannot be inferred without documentation.

Collect kernel and service logs before restarting NICs, clearing conntrack or restarting services because those actions change the evidence.

journalctl --since '2026-07-10 14:00:00' --until '2026-07-10 14:10:00' -k --no-pager
journalctl --since '2026-07-10 14:00:00' --until '2026-07-10 14:10:00' -u your-service.service --no-pager

Look for NETDEV WATCHDOG, link up/down, driver reset, nf_conntrack: table full, OOM and neighbor‑table failures and correlate timestamps with the traffic window.

2. Initial Assessment: Reachability, Path and Business Verification

Establish a baseline with low‑frequency ICMP: ping -n -c 100 -i 0.2 10.30.2.18 Record packet loss, max RTT and jitter.

Examine the path and MTU clues:

tracepath -n 10.30.2.18

tracepath

may show a PMTU hint; encapsulations (VPN, VXLAN, IPsec) can lower the actual usable payload.

Validate the business protocol. HTTP health‑check example:

curl --noproxy '*' -sS -o /dev/null -w 'http_code=%{http_code} connect=%{time_connect} starttransfer=%{time_starttransfer} total=%{time_total}
' --connect-timeout 3 --max-time 10 https://10.30.2.18/healthz

For TLS with a domain name use --resolve instead of -k. TCP port check: nc -vz -w 3 10.30.2.18 443 Success only proves the three‑way handshake, not HTTP/TLS or application success.

3. Command Checks: NIC, TCP, Soft‑Interrupt and conntrack

Inspect NIC counters incrementally (not just cumulative values):

ip -s link show dev eth0
ethtool -S eth0 | rg -i 'drop|discard|error|miss|timeout|buffer|fifo'
nstat -az | rg 'TcpRetransSegs|TcpExtTCPSynRetrans|IpInDiscards|IpOutDiscards'
sleep 60
ip -s link show dev eth0

If RX dropped, rx_missed_errors or driver‑queue counters grow during the failure window, packets reach the NIC receive path but further analysis is needed.

Socket state for the target address:

ss -tin dst 10.30.2.18 dport = :443
ss -tin sport = :443

Growth of bytes_retrans indicates retransmission on that socket. For new connections watch TcpExtTCPSynRetrans; for existing connections watch TcpRetransSegs.

Check receive‑side processing capacity:

mpstat -P ALL 1 5
cat /proc/softirqs
cat /proc/net/softnet_stat

Growth in the second and third columns of softnet_stat often signals insufficient network processing budget (time_squeeze). Do not change net.core.netdev_budget based on a single non‑zero value; require concurrent NIC drops, soft‑interrupt load and business impact.

Read‑only firewall and conntrack inspection:

nft list ruleset
iptables -L -v -n
iptables -t nat -L -v -n
sysctl net.netfilter.nf_conntrack_count net.netfilter.nf_conntrack_max
dmesg -T | rg -i 'nf_conntrack.*table full'

Export the original rule set before any modification and limit changes to the affected source subnet and port.

4. tcpdump Dual‑sided Evidence: Pinpoint Where Packets Disappear

Capture short, filtered traces on both ends with BPF filters, store in a controlled directory, and limit file size to avoid CPU, I/O and data‑leakage risks.

sudo tcpdump -i "${IFACE}" -nn -s 0 -c 2000 -w /var/tmp/source-443.pcap "host ${DST_IP} and tcp port ${PORT}"

sudo tcpdump -i "${IFACE}" -nn -s 0 -c 2000 -w /var/tmp/destination-443.pcap "host ${SRC_IP} and tcp port ${PORT}"

Check directory space and write permission before running:

df -h /var/tmp
test -w /var/tmp && echo writable

For longer captures use -C 50 -W 4 to rotate four 50 MB files.

Interpretation path:

If the source sees SYN but the destination does not, the loss lies between the two capture points; examine routing, ACL, security groups, load balancer, tunnels and intermediate devices.

If the destination sees SYN‑ACK but the source does not receive it, prioritize reverse route, ACL, SNAT and load‑balancer return path.

Repeated same‑sequence numbers with growing bytes_retrans mean the sender did not get ACKs; verify whether the receiver got the data and sent ACKs.

If the destination receives a full request but the application log lacks it, check the listening socket, reverse proxy, TLS/SNI, network namespace and application ingress.

If the destination sends a reply but the source cannot capture it, confirm capture location, return path and NAT state; do not immediately blame a switch.

When checksum offloading is enabled, [bad tcp cksum] in the capture may be harmless; corroborate with the peer capture, NIC error counters and mirror‑port evidence.

5. Container and Kubernetes Scenarios

Container traffic may traverse pod veth, host CNI, overlay tunnel, Service DNAT and node NIC. Use explicit namespace in kubectl commands:

kubectl -n <namespace> get pod <pod-name> -o wide
kubectl -n <namespace> get endpointslice -l kubernetes.io/service-name=<service-name>
kubectl -n <namespace> get networkpolicy
kubectl -n <namespace> exec <pod-name> -- ss -ltnp

Do not assume tcpdump is available inside the pod; if needed run it on the node or in an approved debug container, respecting image provenance and permission scope.

6. Root‑Cause Fix: Change Only the Layer Supported by Evidence

MTU mismatch or PMTU black‑hole

If small requests succeed but large responses retransmit, check MTU:

ip link show dev eth0
tracepath -n 10.30.2.18
ping -M do -s 1472 -c 3 10.30.2.18

Adjust MTU consistently on nodes, CNI or tunnel, record the original MTU, routing and business baseline, test large‑packet workloads, then roll back if needed.

conntrack exhaustion

Only when the conntrack count approaches the max, logs show nf_conntrack: table full, and new connections fail, adjust nf_conntrack_max after backing up sysctl settings and evaluating memory impact. Do not flush the conntrack table, as it would drop existing tracked connections.

Receive‑side soft‑interrupt congestion

Correlate growth in softnet_stat, NIC drops and business anomalies. Adjust RSS/RPS, IRQ affinity, queue count or CPU quota only after review by network and capacity owners, and test on a single node with original settings recorded for rollback.

Unidirectional routing or ACL/NAT

After dual‑sided capture confirms direction, modify firewall, security group, route or NAT cautiously: export original rules, apply changes only to the affected source subnet and port, gray‑scale, and be ready to restore the exported configuration.

7. Verification, Rollback and Post‑mortem

After a fix, replay typical traffic covering the original failure window and compare business success rate, connection errors, P95/P99 latency, TcpRetransSegs delta, NIC/softnet/conntrack deltas and dual‑sided packet captures. Do not rely on a single successful curl; intermittent issues require testing during peak load, same source subnets and connection patterns.

Prepare rollback material beforehand: original routes, policy routes, nftables/iptables rules, sysctl values, MTU, queue/RPS/IRQ settings, CNI/LB configuration versions. Trigger rollback on error‑rate rise, health‑check failure or tail‑latency spikes, stop the spread, restore the single changed item, then re‑verify routing, ports, business functionality and metrics.

In the post‑mortem, separate facts (logs, counters, captures, change records) from inference. Ensure final monitoring includes NIC errors/discards, TCP retransmissions, conntrack usage, soft‑interrupt load and application timeouts, using the actual exporter metric names.