Diagnosing and Resolving Linux Network Packet Loss: A Step‑by‑Step Guide

Understanding Packet Loss

Packet loss occurs when network packets are discarded before reaching the application, and the loss rate (lost packets / total packets) is a key performance metric. High loss degrades TCP performance, causing congestion, retransmissions, increased latency, and reduced throughput.

1. Initial Verification with hping3

# -c sends 10 packets, -S uses TCP SYN, -p sets port 80
hping3 -c 10 -S -p 80 192.168.0.30

HPING 192.168.0.30 (eth0 192.168.0.30): S set, 40 headers + 0 data bytes
len=44 ip=192.168.0.30 ttl=63 DF id=0 sport=80 flags=SA seq=3 win=5120 rtt=7.5 ms
...
--- 192.168.0.30 hping statistic ---
10 packets transmitted, 5 packets received, 50% packet loss
round-trip min/avg/max = 3.0/609.7/3027.2 ms

The output shows 50 % loss and large RTT variance, indicating packet loss, likely due to retransmissions.

2. Where Could Loss Occur?

The author lists potential loss points across the protocol stack:

VM‑to‑VM transmission errors (congestion, line errors)

NIC ring‑buffer overflow

Link‑layer frame errors or QoS drops

IP‑layer routing failures or MTU issues

Transport‑layer port or resource limits

Socket buffer overflow

Application‑level failures

iptables filtering

3. Link‑Layer Check

Using netstat -i and ethtool the author finds no NIC errors:

netstat -i
Kernel Interface table
Iface  MTU  RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR Flg
eth0   100   31     0      0      0      8     0      0      0 BMRU
lo    65536  0      0      0      0      0     0      0      0 LRU

However, a tc netem rule shows a configured 30 % loss:

tc -s qdisc show dev eth0
qdisc netem 800d: root refcnt 2 limit 1000 loss 30%
 Sent 432 bytes 8 pkt (dropped 4, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0

Removing the netem rule with tc qdisc del dev eth0 root netem loss 30% eliminates the artificial loss, but hping3 still reports 50 % loss.

4. Network & Transport Layer Statistics

Running netstat -s reveals TCP errors:

Tcp:
    11 failed connection attempts
    4 segments retransmitted
    11 resets received for embryonic SYN_RECV sockets
    4 TCPSynRetrans
    7 TCPTimeouts

These indicate many handshake failures, yet the root cause remains unclear, prompting further investigation.

5. iptables Inspection

Connection‑track counters are far below their limits, so they are not the cause. The iptables -t filter -nvL output shows two DROP rules with a 30 % random probability:

Chain INPUT (policy ACCEPT 25 packets, 1000 bytes)
 6 240 DROP all -- * * 0.0.0.0/0 0.0.0.0/0 statistic mode random probability 0.29999999981

Chain OUTPUT (policy ACCEPT 15 packets, 660 bytes)
 6 264 DROP all -- * * 0.0.0.0/0 0.0.0.0/0 statistic mode random probability 0.29999999981

Deleting these rules with:

iptables -t filter -D INPUT -m statistic --mode random --probability 0.30 -j DROP
iptables -t filter -D OUTPUT -m statistic --mode random --probability 0.30 -j DROP

still leaves packet loss unchanged.

6. HTTP Request Failure and tcpdump

Even after the above fixes, a curl --max-time 3 http://192.168.0.30 times out. Capturing traffic with tcpdump -i eth0 -nn port 80 shows the three‑way handshake succeeds, but a FIN packet appears three seconds later, indicating the client closed the connection after the curl timeout.

14:40:00.589235 IP 10.255.255.5.39058 > 172.17.0.2.80: Flags [S], ...
14:40:00.589277 IP 172.17.0.2.80 > 10.255.255.5.39058: Flags [S.], ...
14:40:00.589894 IP 10.255.255.5.39058 > 172.17.0.2.80: Flags [.], ...
14:40:03.589352 IP 10.255.255.5.39058 > 172.17.0.2.80: Flags [F.], ...

The author suspects an MTU mismatch because netstat -i shows the NIC MTU is only 100, far below the Ethernet default of 1500.

netstat -i
Iface  MTU  RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR Flg
eth0   100   157    0     344   0      94    0      0      0 BMRU

Adjusting the MTU to 1500 resolves the issue: ifconfig eth0 mtu 1500 After the change, curl returns the expected Nginx HTML page, confirming that packet loss was fully eliminated.

Conclusion

The systematic approach—starting with simple SYN probes, examining link‑layer statistics, removing artificial netem loss, checking firewall DROP rules, and finally correcting an MTU setting—demonstrates how to pinpoint and fix Linux network packet loss across multiple layers.