How to Diagnose and Fix UDP Packet Drops on Linux

Author: CloudDeveloper – Original article: https://cizixs.com/2018/01/13/linux-udp-packet-drop-debug/

Linux UDP packet‑receive flow

Physical network packets arrive at the NIC.

The NIC driver copies packets into a ring buffer using DMA, without CPU involvement.

The kernel reads packets from the ring buffer, processes IP and UDP/TCP logic, and places them into the application’s socket buffer.

The application reads packets from the socket buffer for processing.

Confirm UDP packet loss

Check NIC counters for errors or drops: ethtool -S eth0 Look for non‑zero bad or drop fields. Also use: ifconfig eth0 and examine the RX and TX statistics.

System‑wide UDP statistics can be obtained with: netstat -s --udp Key fields to watch: packet receive errors – non‑zero and increasing indicates kernel‑level UDP loss. packets to unknown port received – usually harmless, caused by services not listening. receive buffer errors – indicates the UDP receive buffer is too small.

NIC or driver drops

If ethtool -S eth0 shows non‑zero rx_*_errors, the NIC hardware or driver is likely at fault and the vendor should be consulted.

Ring‑buffer statistics can be inspected with: ethtool -g eth0 Typical output:

Ring parameters for eth0:
RX: 4096 (max) / 256 (current)
TX: 4096 (max) / 256 (current)

Increase the receive ring size if needed:

ethtool -G eth0 rx 8192

System‑level causes

UDP checksum errors : corrupted packets are discarded; disabling checksum verification can be done via socket options if appropriate.

Firewall : a firewall that drops UDP will block all packets; verify rules when loss is severe.

Insufficient UDP buffer size : Linux stores incoming packets in a receive buffer whose limits are defined by /proc/sys/net/core/rmem_max and /proc/sys/net/core/rmem_default. Increase them with:

sysctl -w net.core.rmem_max=26214400   # 25 MiB

Persist the change in /etc/sysctl.conf for future reboots.

netdev_max_backlog : controls how many packets the kernel can queue after the NIC driver. Raise it, e.g.:

sysctl -w net.core.netdev_max_backlog=2000

Application‑level drops

Each socket has its own buffer size. Increase it when creating the socket, for example:

uint64_t receive_buf_size = 20*1024*1024; // 20 MiB
setsockopt(socket_fd, SOL_SOCKET, SO_RCVBUF, &receive_buf_size, sizeof(receive_buf_size));

If you cannot modify the source, look for configuration options in the application or raise an issue with the developers. Larger socket buffers reduce loss but consume more memory.

Processing packets asynchronously also helps avoid loss caused by slow consumer loops.

Tracing where packets are dropped

Use dropwatch to see kernel functions that drop packets:

# dropwatch -l kas
1 drops at tcp_v4_do_rcv+0xcd
10 drops at tcp_v4_rcv+0x80
...

Capture traffic with tcpdump for a specific UDP port: tcpdump -i eth0 udp port 2020 -s0 -XX -nn Linux perf can monitor the kfree_skb event, which is invoked when a packet is freed:

sudo perf record -g -a -e skb:kfree_skb
sudo perf script

Summary

UDP is unreliable by design; use TCP for loss‑sensitive applications or add reliability at the application layer.

When packet loss appears, first check system load; high CPU, memory, or I/O pressure can cause drops.

If the load is normal, examine NIC counters, ring buffer sizes, and sysctl parameters to eliminate hardware or kernel bottlenecks.

Adjust both system‑wide and per‑socket buffer sizes to accommodate high‑throughput traffic.

Process packets asynchronously to keep the receive path fast.