Kubernetes IPVS Estimation Timer: Root Cause of Network Jitter and Livepatch Fix

1. Introduction

Qutoutiao's containerization has been running for over a year, with nearly 1000 services containerized and a micro‑service cluster of more than a thousand nodes. As the number of containerized services and the cluster size grew, beyond normal API‑Server and Scheduler tuning we encountered network jitter caused by the IPVS load‑balancing module in Kubernetes. This article summarizes the analysis, troubleshooting and solution.

Cluster and OS versions:

Alibaba Cloud ACK 1.14.8 with CNI plugin terway in terway‑eniip mode;

CentOS 7.7.1908, kernel 3.10.0‑1062.9.1.el7.x86_64;

2. Network Jitter Issue

Newly deployed service A showed occasional 999 ms latency when calling downstream service B within the same cluster, with maximum latency up to 200 ms under low QPS. The calls use gRPC with Consul service discovery.

Initial investigations ruled out abnormal registration nodes, HTTP latency, and VM‑level issues, narrowing the problem to the underlying network on the host of service B.

Ping tests between host A and host B reproduced a latency pattern consistent with the gRPC jitter, allowing us to focus on low‑level network packet analysis.

# 主机 B 中的 Pod IP 地址
# ip route|grep 172.23.14.144
172.23.14.144 dev cali95f3fd83a87 scope link

Ping between the host NIC (eth1) and the container network (cali‑xxx) showed jitter up to 133 ms, while host‑to‑host ping remained stable.

3. Problem Analysis

3.1 Kernel packet processing flow

Network drivers deliver packets via hardware interrupts (ISR). Linux splits interrupt handling into a top half (quick) and a bottom half (softirq) processed by the ksoftirqd kernel thread.

Softirq types include NET_TX_SOFTIRQ (transmit) and NET_RX_SOFTIRQ (receive). The receive path eventually calls net_rx_action(), which invokes ip_rcv() and then icmp_rcv() for ICMP packets.

3.2 Softirq investigation

Using BCC's traceicmpsoftirq.py we observed that the jitter spikes correspond to ksoftirqd/0 handling the packets on CPU 0. Further tests showed that pinging the container IP always landed on CPU 0, while pinging the host IP used CPU 19, indicating an RPS (Receive Packet Steering) configuration that hashes packets to a fixed CPU.

RPS distributes network‑receive processing across CPUs; the hash is based on packet headers (e.g., source IP).

We verified the IPVS timer function estimation_timer runs on CPU 0 and consumes up to 119 ms per iteration, blocking other softirqs.

3.3 IPVS estimation_timer

The timer is created in ip_vs_estimator_net_init() and scheduled every 2 seconds to walk all IPVS rules in the host network namespace. In a large production cluster with ~30 000 rules, the traversal takes >100 ms, causing the observed jitter.

static void estimation_timer(struct timer_list *t)
{
    spin_lock(&ipvs->est_lock);
    list_for_each_entry(e, &ipvs->est_list, list) {
        /* statistics collection */
    }
    spin_unlock(&ipvs->est_lock);
    mod_timer(&ipvs->est_timer, jiffies + 2*HZ);
}

4. Solution

4.1 Mitigation options

Disable the IPVS timer on the CPUs selected by RPS.

Patch the IPVS driver to remove the timer (requires kernel reload).

Move the statistics walk to a dedicated kernel thread.

We chose a live‑patch approach using Red Hat’s kpatch to replace estimation_timer with a no‑op implementation, avoiding a reboot.

4.2 kpatch livepatch workflow

4.2.1 Build kpatch

$ git clone https://github.com/dynup/kpatch.git
$ source test/integration/lib.sh
$ sudo kpatch_dependencies
$ cd kpatch
$ make
$ sudo make install

4.2.2 Create kernel source patch

# diff -u net/netfilter/ipvs/ip_vs_est.c net/netfilter/ipvs/ip_vs_est.c > ip_vs_timer_v1.patch

4.2.3 Build and apply livepatch

# /usr/local/bin/kpatch-build ip_vs_timer_v1.patch --skip-gcc-check --skip-cleanup -r /root/kernel-3.10.0-1062.9.1.el7.src.rpm
# /usr/local/sbin/kpatch load livepatch-ip_vs_timer_v1.ko

Kernel logs confirm the patch is active:

$ dmesg -T
[...]
livepatch: enabling patch 'livepatch_ip_vs_timer_v1'
hotfix estimation_timer patched

4.3 kpatch considerations

The livepatch module must match the exact kernel version.

To persist across reboots, install the patch with kpatch install and enable the kpatch service.

5. Summary

The network jitter investigation spanned application‑level gRPC calls, kernel softirq handling, and IPVS timer behavior. BPF tools such as traceicmpsoftirq.py and softirqs.py were essential for pinpointing the culprit. Applying a kpatch live‑patch to neutralise estimation_timer eliminated the jitter without downtime, demonstrating the power of kernel live‑patching for production‑grade troubleshooting.