How to Diagnose and Fix Network Packet Loss on Linux Servers

Introduction

Packet loss is a frequent network issue that can be observed when a ping to a remote host fails or returns incomplete data, indicating that some packets are being dropped during transmission.

What Is Packet Loss?

Data on the Internet is transmitted in packets measured in bytes. Due to factors such as network device performance or link quality, the received data may be smaller than the sent data, resulting in loss.

Packet Transmission Principle

Application data is encapsulated with a TCP header, then an IP header, and finally a 14‑byte MAC header to form a frame. The NIC adds synchronization bits and CRC before sending the packet onto the wire. Reception reverses this process: the NIC checks CRC, strips the MAC header, copies the frame to a ring buffer, notifies the kernel, and the protocol stack decodes the packet for the application.

Core Troubleshooting Idea

After understanding send/receive mechanics, packet loss can be attributed to three major components: the NIC hardware, the NIC driver, and the kernel protocol stack. The analysis follows a bottom‑up layered approach, examining each layer’s possible loss scenarios and the relevant diagnostics.

Overview of Loss Scenarios

Hardware NIC loss

NIC driver loss

Ethernet link‑layer loss

IP‑layer loss

Transport‑layer UDP/TCP loss

Application‑layer socket loss

Hardware NIC Loss

Ring Buffer Overflow

When the NIC writes incoming frames to its internal ring buffer faster than the kernel consumes them, the buffer fills and new packets are dropped.

$ ethtool -S eth0 | grep rx_fifo

$ cat /proc/net/dev

Solution: increase NIC hardware receive and transmit buffers. $ ethtool -G eth0 rx 4096 tx 4096 Port Negotiation Loss

$ ethtool -S eth1

$ ethtool eth1

Solution: re‑negotiate or force the speed.

$ ethtool -r eth1

$ ethtool -s eth1 speed 1000 duplex full autoneg off

Flow Control Loss $ ethtool -S eth1 | grep control Solution: disable flow control.

$ ethtool -A eth1 autoneg off</code><code>$ ethtool -A eth1 tx off</code><code>$ ethtool -A eth1 rx off

MAC‑Address Mismatch

If the destination MAC does not match the NIC’s address (e.g., stale ARP entries), packets are dropped.

$ tcpdump -i eth0 -e

$ arp -n

Solution: refresh ARP tables or set static entries.

Other NIC Anomalies

Check firmware version and replace faulty cables.

$ ethtool -i eth1

$ ethtool -S eth0

NIC Driver Loss

Key counters:

RX errors – includes CRC, frame sync, FIFO overruns, etc.

RX dropped – packets discarded after entering the ring buffer due to memory shortage.

RX overruns – kernel cannot keep up with NIC interrupts. $ cat /proc/net/softnet_stat Solution: increase net.core.netdev_max_backlog and balance interrupts. $ sysctl -w net.core.netdev_max_backlog=2000 Adjust RSS queues, interrupt affinity, and enable/disable interrupt coalescing as needed:

$ ethtool -x eth0</code><code>$ ethtool -X eth0 weight</code><code>$ ethtool -c eth0</code><code>$ ethtool -C eth0 adaptive-rx on

Kernel Protocol‑Stack Loss

Ethernet Link‑Layer

Usually covered by NIC diagnostics above.

ARP/Neighbour Loss

Parameters arp_ignore and arp_filter control how the kernel replies to ARP requests. Misconfiguration can cause drops.

$ sysctl -a | grep arp_ignore

$ sysctl -a | grep arp_filter

Set appropriate values based on the environment.

ARP table overflow also leads to drops. Check statistics and increase gc_thresh* limits.

$ sysctl -w net.ipv4.neigh.default.gc_thresh1=1024</code><code>$ sysctl -w net.ipv4.neigh.default.gc_thresh2=2048</code><code>$ sysctl -w net.ipv4.neigh.default.gc_thresh3=4096

IP‑Layer Loss

Common causes:

Incorrect IP address configuration (e.g., missing loopback address).

Missing local routing entries.

Incorrect routing tables.

$ ip a add 1.1.1.1 dev eth0</code><code>$ ip r add local 1.1.1.1 dev eth0 table local

Adjust rp_filter if reverse‑path filtering drops legitimate packets.

$ sysctl -w net.ipv4.conf.all.rp_filter=2

Firewall Drops

$ iptables -nvL | grep DROP

Modify rules to allow required traffic.

Connection‑Tracking Table Overflow

When the nf_conntrack table fills, packets are dropped. $ cat /proc/sys/net/netfilter/nf_conntrack_max Increase the table size and reduce timeout values.

$ sysctl -w net.netfilter.nf_conntrack_max=3276800</code><code>$ sysctl -w net.netfilter.nf_conntrack_tcp_timeout_established=1200

Transport‑Layer Loss

TCP Issues

Typical problems include TIME_WAIT overflow, SYN‑flood, window‑size mismatches, PAWS, and re‑ordering. $ sysctl -a | grep tcp_max_tw_buckets Solutions: enable tcp_tw_reuse, increase tcp_max_tw_buckets, adjust tcp_syncookies, and tune tcp_max_syn_backlog.

$ sysctl -w net.ipv4.tcp_max_syn_backlog=4096

UDP Issues

UDP is connection‑less; loss is often due to buffer exhaustion. $ netstat -s | grep "receive buffer errors" Increase UDP memory settings:

$ sysctl -w net.ipv4.udp_mem="..."</code><code>$ sysctl -w net.ipv4.udp_rmem_min=...

Application‑Layer Socket Loss

Socket buffer sizes may be insufficient.

$ sysctl -w net.core.rmem_max=67108864</code><code>$ sysctl -w net.core.wmem_max=33554432

Adjust application‑level socket options ( setsockopt) accordingly.

Tools for Diagnosis

dropwatch monitors kernel drop events and prints stack traces. $ dropwatch -d tcpdump captures packets for offline analysis. $ tcpdump -i eth0 -w capture.pcap Wireshark or tshark can be used to further dissect the captures.