How to Diagnose Intermittent Packet Loss Using mtr, ss, and tcpdump

Overview

Intermittent packet loss can be hard to reproduce, but a structured troubleshooting flow helps isolate the root cause. The method starts at the application layer, moves through the TCP stack, inspects kernel and NIC statistics, validates the network path with mtr, and finally captures packets with tcpdump for detailed analysis.

Step‑by‑Step Process

Confirm Application Symptoms – Search logs for timeout, connection reset, or read/write errors to ensure the issue is network‑related.

Inspect TCP Connections with ss – Run ss -nti state established to view bytes_retrans, retrans, Send‑Q, and Recv‑Q. Persistent non‑zero values indicate retransmissions or queue buildup.

Check Kernel Network Counters – Examine /proc/net/snmp and /proc/net/netstat for fields such as InDiscards, OutDiscards, RetransSegs, and TCPTimeouts. Rising counters point to drops or retransmissions at the kernel level.

Validate the Path with mtr – Use mtr -r -c 100 <target_ip> to generate a loss report per hop. Identify the hop where loss spikes, distinguishing local, intermediate, or remote issues.

Capture Traffic with tcpdump – Run tcpdump -i eth0 -nn host <target_ip> -w capture.pcap. Analyze the pcap for duplicate ACKs, out‑of‑order segments, zero windows, or RST packets.

Analyze NIC Statistics – Use ip -s link show eth0 and ethtool -S eth0 to check rx_dropped, tx_dropped, rx_errors, and ring‑buffer sizes. Adjust with ethtool -G eth0 rx <size> tx <size> if needed.

Review Kernel Parameters – Verify net.ipv4.tcp_rmem, net.ipv4.tcp_wmem, net.core.somaxconn, and net.core.netdev_max_backlog. Tune them with sysctl -w or persist in /etc/sysctl.conf.

Check Firewall and Connection Tracking – List iptables rules ( iptables -L -n -v) and connection‑track table size ( sysctl net.netfilter.nf_conntrack_max). Reduce drops or increase limits as appropriate.

Remote End Verification – If local checks are clean, repeat the above steps on the remote host and examine its logs, NIC stats, and firewall.

Finalize Fixes – Based on the identified layer, apply one of the following: increase NIC ring buffers, raise somaxconn, enlarge netdev_max_backlog, adjust TCP buffers, switch congestion control (e.g., to bbr), or modify firewall rules.

Verification

After changes, monitor the same counters (ss retrans, kernel InDiscards / OutDiscards, mtr loss rate) to confirm they remain stable or improve. Use watch -n 1 "cat /proc/net/snmp | grep Tcp" and repeat mtr runs.

Rollback Plan

Document original values before any modification. Revert temporary sysctl -w changes with a reboot or by restoring the saved values. For persistent edits, edit /etc/sysctl.conf back to the original settings and run sysctl -p. Restore previous NIC ring sizes with ethtool -G eth0 rx <old> tx <old>. If firewall rules were altered, reload the saved iptables backup using iptables-restore < /path/to/backup.txt.