Why Is My Linux Server Dropping Packets? A Step‑by‑Step Debug Guide

1. Where packet loss can occur

Packet loss may happen at any layer of the network stack, from the VM link to the application layer. Common sources include:

Transmission errors between VMs (congestion, line errors)

Ring‑buffer overflow on the NIC

Link‑layer frame checksum failures or QoS drops

IP‑layer routing failures or MTU oversize

Transport‑layer port not listening or kernel limits

Socket buffer overflow

Application‑level crashes

iptables filtering rules

2. Link‑layer analysis

When the NIC drops packets due to buffer overflow, Linux records error counters. Use ethtool or netstat -i to view them. netstat -i Sample output (truncated):

Kernel Interface table
Iface   MTU RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR Flg
eth0   100   31     0      0      0    0      8      0      0    BMRU

Key fields:

RX‑OK : total received packets

RX‑ERR : total receive errors

RX‑DRP : packets dropped after entering the ring buffer

RX‑OVR : ring‑buffer overflow drops

TX equivalents represent sending statistics.

The example shows no NIC errors, but a tc rule is present: tc -s qdisc show dev eth0 Output indicates a netem qdisc with loss 30%, causing 4 of 8 sent packets to be dropped.

3. Network and transport layer analysis

Linux provides per‑protocol statistics via netstat -s. Focus on error, drop, and retransmission counters. netstat -s Relevant excerpt:

Tcp:
  11 failed connection attempts
  4 segments retransmitted
  11 resets received for embryonic SYN_RECV sockets
  4 SYN retransmissions
  7 timeouts

These figures show many TCP‑level failures, mainly during the three‑way handshake.

4. iptables inspection

iptables (Netfilter) rules can also drop packets. List the filter table with counters: iptables -t filter -nvL Sample output (truncated):

Chain INPUT (policy ACCEPT 25 packets, 1000 bytes)
 pkts bytes target prot opt in out source destination
   6 240 DROP all -- 0.0.0.0/0 0.0.0.0/0 statistic mode random probability 0.30

Chain OUTPUT (policy ACCEPT 15 packets, 660 bytes)
 pkts bytes target prot opt in out source destination
   6 264 DROP all -- 0.0.0.0/0 0.0.0.0/0 statistic mode random probability 0.30

Both INPUT and OUTPUT chains contain a statistic rule that randomly drops 30 % of packets – the likely culprit for the observed loss.

5. Removing the offending tc and iptables rules

Delete the netem rule: tc qdisc del dev eth0 root netem loss 30% Delete the iptables DROP rules:

iptables -t filter -D INPUT -m statistic --mode random --probability 0.30 -j DROP
iptables -t filter -D OUTPUT -m statistic --mode random --probability 0.30 -j DROP

6. Re‑testing with hping3

After rule removal, run the same hping3 test: hping3 -c 10 -S -p 80 192.168.0.30 Result shows 0 % packet loss (10/10 packets transmitted and received) and stable RTT.

7. Curl test and hidden MTU issue

Even though SYN packets succeed, curl still times out: curl --max-time 3 http://192.168.0.30 Inspect netstat -i again; the NIC reports RX‑DRP 344 and an MTU of only 100 bytes on eth0. The default Ethernet MTU is 1500 bytes, so the tiny MTU truncates larger TCP segments (e.g., HTTP GET requests), causing drops.

8. Fixing MTU

Set the correct MTU: ifconfig eth0 mtu 1500 Re‑run the curl command: curl --max-time 3 http://192.168.0.30/ The response HTML is returned, confirming that packet loss is resolved.

9. Final verification

Run netstat -i once more; the drop counters return to zero and the MTU shows 1500. The system now handles both SYN and full HTTP traffic without loss.