Understanding Linux NIC Packet Reception and How to Fix Drop Issues

Linux NIC Packet Reception Process

The NIC receives a data packet.

The packet is moved from the NIC hardware cache to server memory via DMA (CPU‑free).

A hardware interrupt notifies the CPU.

The CPU triggers a soft‑interrupt to let the kernel handle the packet.

The packet traverses the TCP/IP stack.

The application reads the data from the socket buffer with read().

Diagnosing Packet Loss

# ifconfig eth0
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
        inet 10.5.224.27 netmask 255.255.255.0 broadcast 10.5.224.255
        inet6 fe80::5054:ff:fea4:44ae prefixlen 64 scopeid 0x20<link>
        ether 52:54:00:a4:44:ae txqueuelen 1000 (Ethernet)
        RX packets 9525661556 bytes 10963926751740 (9.9 TiB)
        RX errors 0 dropped 0 overruns 0 frame 0
        TX packets 8801210220 bytes 12331600148587 (11.2 TiB)
        TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

Key fields:

RX errors : total receive errors (e.g., too‑long frames, CRC errors, FIFO overruns, missed packets).

RX dropped : packets entered the ring buffer but were discarded during copy to memory due to insufficient resources.

RX overruns : FIFO overrun; packets are dropped before reaching the ring buffer because the NIC’s FIFO is full, often caused by high CPU load or uneven interrupt distribution.

RX frame : misaligned frames.

Difference between dropped and overruns :

Dropped : packet reached the NIC’s receive FIFO and was being processed when the system ran out of memory or other resources, so Linux discarded it.

Overruns : packet never entered the FIFO because the FIFO was already full; the NIC hardware dropped it, usually indicating that the kernel cannot keep up with interrupt handling.

Typical statistics screenshots (normal vs. abnormal) are shown below:

Ring Buffer Overflow

When the ring buffer is too small, the arrival rate of packets exceeds the kernel’s consumption rate, causing the buffer to fill quickly and new packets to be dropped.

Check overflow statistics with:

# ethtool -S ens2 | grep fifo
rx_fifo_errors: 0
tx_fifo_errors: 0

Adjust the ring buffer size: # ethtool -G eth0 rx 4096 tx 4096 Output shows the preset maximums and current hardware settings.

Interrupt Handling and Multi‑Queue NICs

Two types of interrupts exist:

Hard interrupt : generated by hardware (NIC, disk, keyboard) and sent directly to the CPU.

Soft interrupt : generated by the kernel after the hard interrupt to process the packet.

When a NIC receives a packet, it raises a hard interrupt; the CPU stops its current task, notifies the kernel, which then runs a soft‑interrupt handler to copy the packet from the NIC cache to memory before the FIFO overflows.

Modern NICs use multiple queues (RSS – Receive Side Scaling) to distribute packets across CPUs, improving parallel processing. If RSS is unavailable, Receive Packet Steering (RPS) provides a software‑based similar mechanism.

Configure IRQ affinity to bind specific interrupts to chosen CPUs:

# echo 6 > /proc/irq/41/smp_affinity   # binds IRQ 41 to CPU1 and CPU2 (bitmask 0x6)

SoftIRQ Statistics and Netdev Backlog Tuning

View per‑CPU softirq counts:

# cat /proc/softirqs
CPU0  CPU1
HI: 1 0
TIMER: 1650579324 3521734270
NET_TX: 10282064 10655064
NET_RX: 3618725935 2446
...

NET_RX

indicates the number of packets that triggered a softirq on each CPU. Uneven distribution may suggest the NIC lacks multi‑queue support; enabling RPS can balance the load.

Enable RPS by setting the CPU mask for a receive queue:

# echo f > /sys/class/net/eth0/queues/rx-0/rps_cpus   # uses CPUs 0‑3 (mask 0xf)

The kernel parameter net.core.netdev_max_backlog defines the size of the backlog queue before the protocol stack processes packets. The default is 1000; increase it to reduce drops caused by backlog overflow: # sysctl -w net.core.netdev_max_backlog=2000 Monitor the backlog with /proc/net/softnet_stat to verify that the adjustments have taken effect.