Redis Timeouts Traced to CPU Softirq Scheduling – Full End‑to‑End Investigation

Problem – Intermittent Redis Timeout

On Kubernetes node docker97.xxx.qihoo.net some pods experience occasional Redis request latency spikes that exceed the timeout threshold. Application logs show sporadic long response times and timeouts, while Redis metrics (CPU, connections, QPS) remain normal.

Verification – Excluding the Network Layer

Inside the pod network namespace the command: nc -z -v 10.220.193.137 25747 shows most connections succeed instantly, but a few attempts take noticeably longer. An MTR capture shows no persistent packet loss, indicating the issue is not a classic link failure but an unstable processing capability on the node.

Shift of Perspective – CPU / Interrupt / Softirq Layer

When application behavior is normal and the network shows no hard loss, the investigation should focus on CPU, interrupt, and soft‑interrupt handling capacity. Redis, as a high‑frequency short‑connection workload, is highly sensitive to soft‑interrupt scheduling latency: packets arrive at the NIC, but the time the CPU processes them directly translates into request latency.

Deep Dive – Tracing the Softirq Culprit

4.1 CPU and Interrupt Overview

Run:

mpstat -P ALL 1
mpstat -I SCPU 1 5

to observe that %soft (soft‑interrupt usage) on a few cores is significantly higher than on others, indicating uneven soft‑interrupt distribution.

Inspect interrupt statistics:

cat /proc/interrupts | less -S
cat /proc/softirqs | less -S

Network‑related IRQ/softirq traffic is concentrated on a small set of CPUs.

Aggregate per‑CPU interrupt counts:

awk '{for(i=2;i<=NF;i++)a[i]+=$i} END{for(i in a)print "CPU"i-2":",a[i]}' /proc/interrupts

This confirms that network IRQ/softirq is heavily skewed toward a few cores.

4.2 NIC and NUMA Topology

Identify the NIC model and PCIe location:

# ethtool -i eth2
driver: ixgbe
version: 5.10.134-14.an8.x86_64
firmware-version: 0x80000625
bus-info: 0000:84:00.0

Check the NUMA node of the NIC: cat /sys/class/net/eth2/device/numa_node The output shows the NIC is attached to NUMA node 0.

Inspect NIC queue‑to‑CPU affinity:

grep . /proc/irq/*/smp_affinity_list

If interrupts are processed on CPUs belonging to a different NUMA node, remote memory accesses are introduced, causing latency jitter.

Generate a full NUMA + PCIe topology diagram:

# yum install -y hwloc hwloc-gui
# sudo lstopo --whole-system --whole-io --rect --fontsize 9 --gridsize 8 /root/topo.svg

4.3 irqbalance – The Well‑Intentioned Troublemaker

The Linux service irqbalance periodically reads /proc/interrupts and attempts to distribute interrupts evenly across CPUs. In high‑throughput, low‑latency workloads such as Redis, moving interrupts can cause:

Cache miss : each migration invalidates CPU caches.

NUMA remote access : interrupts may be handled on a CPU of a remote NUMA node, incurring memory latency.

Softirq processing delay : packets wait longer in the NIC queue before a CPU picks them up.

This creates a classic trade‑off between stability and throughput.

Check the service status:

systemctl status irqbalance

Root Cause – Misaligned Interrupt & Softirq Affinity

The chain of events:

Multi‑queue NIC + irqbalance automatic migration

Interrupts spread across NUMA nodes

Softirq (NET_RX) concentrates on a few CPUs, causing processing delay

Pod network requests become stuck in the kernel

Application layer observes intermittent Redis request timeouts

Solution – Targeted Remediation

Design principle : interrupt handling should prioritize stability rather than pure load‑balancing; IRQ/softirq, CPU, and NUMA must be coordinated, and network I/O deserves high priority.

Solution 1 – Align NIC Queue Count with NUMA

ethtool -L eth0 combined 20

Set the number of queues roughly equal to the number of CPUs on NUMA node 0, preventing queues from being scheduled on remote nodes.

Solution 2 – Increase NIC Ring Buffer

ethtool -G eth0 rx 2048 tx 2048

Boost burst‑traffic handling capacity, reducing RX packet loss and back‑pressure.

Solution 3 – Change irqbalance CPU‑Binding Strategy

# 1. Stop the service
systemctl stop irqbalance

# 2. Use NUMA‑aware IRQ binding
# Verify binding
cat /etc/sysconfig/irqbalance
IRQBALANCE_BANNED_CPUS=3fffffff,ffff3fff,ffffffff

# 3. Restart the service
systemctl start irqbalance

Reserve specific CPUs for the koordinator scheduler so business pods do not compete for the same cores:

kubectl get node -o json | jq -r '.items[] | [.metadata.name, (.metadata.annotations["node.koordinator.sh/reservation"] // "")] | @tsv' | grep "docker042.cloud.bjmd.qihoo.net"
# Example output
docker042.cloud.bjmd.qihoo.net  {"resources":{"memory":"162143134516"},"reservedCPUs":"46,47,94,95"}

Validation and Methodology Consolidation

Validation metrics :

Redis request timeouts disappear.

%softirq drops noticeably.

Interrupt distribution stabilizes, no frequent migrations.

Pod network response‑time variance shrinks significantly.

Methodology takeaway : when the following pattern appears in production, jump directly to IRQ/softirq/NUMA investigation:

Intermittent application network timeouts.

No obvious bottleneck in Redis or MySQL.

MTR shows no stable packet loss.

CPU %soft is abnormally high.

Additional Optimization

Isolate kubelet and containerd processes onto dedicated CPU cores to avoid contention with business pods, further reducing unpredictable scheduling jitter.

Reference: https://cloud.tencent.com/developer/article/2383533