Boosting Kubernetes Service Performance: Inside Tencent’s IPVS‑BPF Optimization

1. Current Service Modes and Their Limitations

Kubernetes Service provides intra‑cluster communication and load balancing via three main implementations: userspace, iptables, and IPVS. IPVS offers the best performance but still relies on nf_conntrack for SNAT, introducing considerable CPU and latency overhead due to its complex state machine.

In iptables mode, scalability suffers because each new rule requires traversing and modifying the entire rule set, resulting in O(n²) control‑plane complexity and O(n) data‑plane processing.

IPVS uses hash tables for O(1) service lookup, yet its dependence on nf_conntrack for SNAT creates a performance bottleneck.

2. IPVS‑BPF Design Overview

The Tencent TKE team created an IPVS‑BPF mode that completely bypasses nf_conntrack and implements SNAT with eBPF. The core ideas are:

Introduce a switch in the IPVS kernel module to toggle between native IPVS logic and the new IPVS‑BPF logic.

Move the IPVS hook from LOCAL_IN to PREROUTING so that Service requests skip nf_conntrack.

Update connection‑creation and deletion code to add or remove session entries in an eBPF map.

Attach eBPF SNAT code to the qdisc layer, using the session map to perform address translation.

Additional handling for ICMP and packet fragmentation is also provided. The solution keeps the code size manageable: roughly 500 lines of BPF code and 1,000 lines of IPVS modifications.

3. Performance Evaluation

Measurements were performed with perf for CPU counters and with wrk (short‑connection) and iperf (long‑connection) workloads. The test environment used a single‑core LB node and an eight‑core backend node to avoid client‑side bottlenecks.

Results:

NodePort short‑connection throughput increased by 64% and p99 latency dropped by 47%.

ClusterIP short‑connection throughput increased by 40% and p99 latency dropped by 31%.

Long‑connection bandwidth (iperf) improved by 22%.

Average CPU instructions per request decreased by 38%, the primary cause of the speedup.

CPI rose modestly (~16%), indicating higher per‑cycle work that warrants further study.

4. Additional Optimizations, Limitations, and Future Work

During development, the team also fixed issues such as low performance when conn_reuse_mode=1, occasional DNS delays, and external IP health‑check failures. Limitations include inability for a pod to reach its own Service (requests are redirected to other pods) and the need for a whitelist to enable the feature.

Future directions involve leveraging Cilium‑style eBPF techniques to further accelerate ClusterIP traffic and investigating the root cause of the CPI increase.

5. Enabling IPVS‑BPF in Tencent Cloud TKE

To activate the mode, create a cluster in the TKE console, go to Advanced Settings → Kube‑proxy Mode, and select ipvs-bpf. The feature currently requires a whitelist request via the official application page.