Mastering NIC Ring Buffers: How Linux Handles Packets and Boosts Performance

1. NIC Packet Processing Flow

Diagram of the NIC packet handling process:

Explanation of the dashed steps in the diagram:

1. DMA writes incoming packets into sk_buff structures, possibly spanning multiple buffers; sk_buff follows FIFO order.

2. After DMA finishes, the NIC triggers an interrupt (IRQ) via the NIC Interrupt Handler.

3. The NIC driver registers a poll function.

4. The poll function checks data, merging multiple sk_buff if a packet is split.

5. The poll function hands the sk_buff to the upper network stack.

Complete Process

1. On system boot, the NIC is initialized and memory is allocated for the Ring Buffer.

2. Each slot in the Ring Buffer initially holds a Packet Descriptor pointing to a ready sk_buff.

3. DMA writes packets into sk_buff; used buffers change to the "used" state.

4. After DMA reads, the NIC triggers an IRQ.

5. The NIC driver registers the poll function.

6. The poll function merges split buffers and delivers them to the network stack.

7. The poll function cleans up sk_buff and resets descriptors to point to newly allocated buffers, marking them ready.

2. Ring Buffer Handling on Multi‑CPU Systems

When packet arrival rate exceeds a single CPU's processing speed, the Ring Buffer can fill and drop new packets. On multi‑core servers, NICs may provide multiple Ring Buffers and distribute IRQs across CPUs, enabling parallel processing and higher throughput. This requires NIC support for Receive Side Scaling (RSS) or multiqueue.

3. Ring Buffer Related Commands

In production, Ring Buffer overflow often causes packet loss. The following commands help monitor and tune the buffers.

3.1 Packet Statistics

[root@test]$ ethtool -S em1 | more
NIC statistics:
    rx_packets: 35874336743
    tx_packets: 35163830212
    rx_bytes: 6337524253985
    tx_bytes: 3686383656436
    rx_broadcast: 15392577
    tx_broadcast: 873436
    rx_multicast: 45849160
    tx_multicast: 1784024

RX denotes received data; TX denotes transmitted data.

3.2 Error and Drop Statistics

[root@test]$ ethtool -S em1 | grep -iE "error|drop"
rx_crc_errors: 0
rx_missed_errors: 0
... (other error counters) ...
rx_fifo_errors: 79270
rx_queue_0_drops: 16669
rx_queue_1_drops: 21522
rx_queue_2_drops: 0
rx_queue_3_drops: 5678
rx_queue_4_drops: 5730
rx_queue_5_drops: 14011
rx_queue_6_drops: 15240
rx_queue_7_drops: 420

The sum of all queue drops equals rx_fifo_errors, indicating Ring Buffer overflow. Adjusting queue distribution or increasing Ring Buffer size can mitigate loss.

3.3 Query Ring Buffer Size

[root@test]$ ethtool -g em1
Ring parameters for em1:
Pre-set maximums:
RX: 4096
RX Mini: 0
RX Jumbo: 0
TX: 4096
Current hardware settings:
RX: 256
RX Mini: 0
RX Jumbo: 0
TX: 256

Maximum RX/TX is 4096; current value is 256. Larger queues reduce loss but increase latency.

3.4 Adjust Ring Buffer Queue Count

[root@test]$ ethtool -l em1
Channel parameters for em1:
Pre-set maximums:
RX: 0
TX: 0
Other: 1
Combined: 8
Current hardware settings:
RX: 0
TX: 0
Other: 1
Combined: 8

Combined = 8 means the NIC uses eight processes to handle network data.

ethtool -L eth0 combined 8

Changes may require a reboot to take effect.

3.5 Adjust Ring Buffer Queue Size

[root@test]$ ethtool -G em1 rx 4096
[root@test]$ ethtool -G em1 tx 4096

Increasing RX/TX queues reduces packet loss at the cost of higher latency.

3.6 Adjust Queue Weights (RSS Indirection Table)

If the NIC supports multiqueue, it distributes packets based on a hash function. The following shows the indirection table for an 8‑queue NIC (128 entries).

[root@test]$ ethtool -x em1
RX flow hash indirection table for em1 with 8 RX ring(s):
    0: 0 0 0 0 0 0 0 0
    8: 0 0 0 0 0 0 0 0
   16: 1 1 1 1 1 1 1 1
   ...
  120: 7 7 7 7 7 7 7 7
RSS hash key:
Operation not supported

The table maps hash values to specific queues; weights must sum to ≤128 (the table size).

3.7 Change Ring Buffer Hash Fields

Packet distribution can be based on different header fields. Example for TCP over IPv4:

[root@test]$ ethtool -n em1 rx-flow-hash tcp4
TCP over IPV4 flows use these fields for computing Hash flow key:
IP SA
IP DA
L4 bytes 0 & 1 [TCP/UDP src port]
L4 bytes 2 & 3 [TCP/UDP dst port]

Other hash configurations can be set with ethtool -N, e.g.:

ethtool -N em1 rx-flow-hash udp4 sdfn

3.8 IRQ Statistics

View /proc/interrupts to monitor per‑CPU IRQ counts, confirming multiqueue and NAPI IRQ coalescing effectiveness.

Reference Links

Linux Receive Packet – Part 1

Ring Buffer Tuning Guide