How to Diagnose and Fix Linux UDP Packet Drops

When UDP packets are lost on a Linux server, the problem can stem from the network card, driver, kernel, or the application. The article first outlines the packet‑receiving flow: the NIC receives frames, the driver DMA‑writes them into a ring buffer, the kernel processes them (IP and UDP layers) and places them into the socket buffer for the application.

Confirm UDP Packet Loss

Use ethtool -S eth0 and look for fields containing bad or drop. Non‑zero values that keep increasing indicate NIC‑level loss. ifconfig also shows RX/TX statistics; non‑zero error or drop counters suggest hardware or driver issues. netstat -s -u displays UDP‑specific counters. Pay attention to packet receive errors, packets to unknown port received, and receive buffer errors.

Network‑Card or Driver Drops

If ethtool -S shows rx_*_errors, the NIC is likely faulty and should be replaced or serviced. The ring buffer size can be inspected with ethtool -g eth0 and increased, e.g., ethtool -G eth0 rx 8192.

Linux System Drops

Common causes include malformed UDP packets, firewall rules, insufficient UDP buffer sizes, and high system load.

UDP Packet Errors

Checksum or length errors cause the kernel to discard packets. Disabling checksum verification via socket options can be used for debugging.

Firewall

Misconfigured firewalls may drop all or some UDP traffic; verify rules before investigating further.

UDP Buffer Size Insufficiency

The kernel’s receive buffer limits are set in /proc/sys/net/core/ (e.g., rmem_max, rmem_default). Increase them with sysctl -w net.core.rmem_max=26214400 or edit /etc/sysctl.conf. Adjust netdev_max_backlog to enlarge the kernel’s packet queue, e.g., sysctl -w net.core.netdev_max_backlog=2000.

High System Load

CPU, memory, or I/O saturation can prevent timely packet processing, leading to drops. Reduce load or scale resources.

Application‑Level Drops

Each socket has its own buffer; set a larger receive buffer when creating the socket, for example:

uint64_t receive_buf_size = 20*1024*1024; // 20 MB
setsockopt(socket_fd, SOL_SOCKET, SO_RCVBUF, &receive_buf_size, sizeof(receive_buf_size));

If you cannot modify the code, look for configuration options in the application or raise an issue with its developers. Larger buffers reduce loss but increase memory usage.

Tracing Drop Locations

Use dropwatch -l kas to monitor kernel drop events and identify the functions where packets are discarded. Example output shows drops in functions like tcp_v4_do_rcv and sk_stream_kill_queues.

Capture packets with tcpdump -i <em>interface</em> udp port 2020 -s0 -XX -nn and use perf record -g -a -e skb:kfree_skb to trace the kfree_skb call that frees dropped skbs.

Summary

UDP is unreliable; use it only when occasional loss is acceptable.

Check system load first; high load often causes drops.

Adjust kernel and socket buffer sizes to handle high‑throughput UDP traffic.

Prefer asynchronous processing in applications to avoid blocking the receive path.

When drops persist, use ethtool, netstat, dropwatch, and perf to pinpoint the failure point.