Why kube-proxy Causes Traffic Imbalance and How to Fix It in Kubernetes

Introduction

This article addresses traffic imbalance problems observed in production and presents solutions.

Kubernetes is a complex system, and network‑related issues are among the most difficult; the imbalance originates from kube-proxy using the default iptables load‑balancing, which forwards packets probabilistically, causing significant bias when long‑lived connections are few.

Scenarios

2.1 Rolling Update Load Imbalance

When the number of connections is relatively stable, during a rolling update the connections on old Pods are gradually broken and re‑connected to newly started Pods; the Pods that start earlier receive more connections, leading to uneven load.

2.2 RR Strategy Load Imbalance

If long‑lived connections have widely varying durations, the default round‑robin (rr) scheduling of ipvs may cause some backend Pods to receive many long‑lasting connections, resulting in higher connection counts on those Pods. The forwarding status can be inspected with ipvsadm -Ln -t CLUSTER-IP:PORT.

2.3 Scaling Failure Issue

When connections are relatively stable, Horizontal Pod Autoscaler (HPA) scaling creates new Pods, but existing long‑lived connections remain bound to the original Pods, leaving the newly created Pods with almost no traffic, causing the old Pods to stay overloaded and scaling to be ineffective.

We found that some Pods have higher connection counts and therefore consume more resources, leading to load imbalance.

Setting kube-proxy’s ipvs mode to lc (Least‑Connection) tends to forward to Pods with fewer connections, but because ipvs state is distributed across nodes and not globally converged, true least‑connection behavior cannot be guaranteed. Using sh (Source Hashing) can help maintain load balance even without global convergence.

Both ipvs and iptables are based on netfilter; their differences are:

ipvs provides better scalability and performance for large clusters.

ipvs supports more complex load‑balancing algorithms (least‑load, least‑connections, weighted, etc.).

ipvs offers server health checks and connection retry features.

Best Practices

Implement automatic reconnection at the business layer to avoid connections being “stuck” on a specific backend Pod, e.g., periodic reconnection or reconnect after a request‑count threshold.

Avoid direct backend requests; instead use a layer‑7 proxy (e.g., nginx ingress or Istio) for gRPC traffic so that requests are split at the request level and load‑balanced.

Set kube-proxy ipvs scheduler to sh (source hashing) to improve global load distribution.