How to Diagnose and Fix Linux Network Packet Loss: A Step‑by‑Step Guide

Background

The article begins by noting that packet loss can occur at any layer of the network stack, from the virtual machine link to the application layer, and lists typical causes for each layer.

Transmission errors between VMs (congestion, line errors)

Ring‑buffer overflow on the NIC

Link‑layer frame check failures or QoS drops

IP‑layer routing failures or MTU oversize

Transport‑layer port not listening or kernel limits

Socket‑buffer overflow

Application‑layer process crashes

iptables filtering

Link‑layer Inspection

When the NIC drops packets due to buffer overflow, Linux records errors in the NIC statistics. The author shows how to view these statistics with netstat -i and explains the meaning of the columns (RX‑OK, RX‑ERR, RX‑DRP, RX‑OVR, TX‑OK, TX‑ERR, TX‑DRP, TX‑OVR).

In the example, the NIC shows no errors, but a tc rule ( qdisc netem loss 30%) is present, sending 8 packets and dropping 4, which explains the observed 30% loss of Nginx response packets.

Removing the netem rule is done with: tc qdisc del dev eth0 root netem loss 30% After deletion, the author reruns the hping3 test and still sees 50% loss, indicating further investigation is needed.

Network and Transport Layers

The author runs netstat -s to view protocol‑level counters. The output shows TCP having 11 failed connection attempts, 4 retransmitted segments, 11 SYN‑RECV resets, 4 SYN retransmissions, and 7 timeouts, while other protocols report no errors.

These numbers suggest many TCP handshake failures, but the exact cause remains unclear, prompting deeper analysis.

iptables Examination

Connection‑tracking limits are checked with sysctl net.netfilter.nf_conntrack_max and net.netfilter.nf_conntrack_count, confirming that conntrack is not the bottleneck (182/262144).

iptables rules are listed with iptables -t filter -nvL. Two DROP rules using the statistic module with a 30% random probability are found in the INPUT and OUTPUT chains, matching all source and destination IPs. These rules are identified as the source of the packet loss.

Both rules are removed with:

iptables -t filter -D INPUT -m statistic --mode random --probability 0.30 -j DROP
iptables -t filter -D OUTPUT -m statistic --mode random --probability 0.30 -j DROP

Verifying with hping3 and curl

After deleting the iptables rules, hping3 shows 0% loss (10/10 packets received). However, a curl request to the Nginx server still times out, prompting packet capture.

Using tcpdump -i eth0 -nn port 80, the author captures traffic and notes that the NIC statistics now report 344 dropped packets on receive (RX‑DRP), while the MTU for eth0 is only 100 bytes.

MTU Adjustment

The small MTU explains why SYN packets (small) succeed while full HTTP GET packets (larger) are dropped. The fix is to set the MTU to the Ethernet default of 1500 bytes: ifconfig eth0 mtu 1500 After the MTU change, a final curl request returns the expected Nginx HTML page, confirming that packet loss has been fully resolved.