How Zalando Achieved 1M RPS with an In‑Process Client‑Side Load Balancer

Background and Challenge

Product‑read API serves millions of requests per second with single‑digit millisecond latency across 25 markets. Each external request is split into up to 100 parallel internal sub‑calls to different product pods. All traffic, including internal calls, previously passed through the shared edge load balancer Skipper. Because batch latency equals the slowest of the 100 hops, tail latency became a critical issue, and the API team could not distinguish latency spikes caused by business logic from those caused by Skipper, which they did not own.

In‑Process Client‑Side Load Balancing

Routing of high‑fan‑out internal traffic was moved into the calling process while Skipper remained responsible for external edge traffic and single‑GET requests.

“Routing decisions for high‑fan‑out internal traffic should happen inside the calling process; Skipper continues to handle edge traffic. We are not replacing Skipper, but upgrading the internal fan‑out path to run as an in‑process client‑side load balancer.” – Conor Gallagher, Senior Principal Engineer, Zalando

Algorithmic Precise Reuse – Avoiding Cache Splits

The client library reproduces Skipper’s consistent‑hash algorithm (xxHash64 with 100 virtual nodes per endpoint). Unit tests verify that both paths route the same key to the identical pod, providing:

Minimized cache changes : Adding or removing a node remaps only about 1/N of the keys, protecting local pod caches.

Hash‑ring consistency : Identical algorithm and virtual‑node count ensure the client and Skipper generate the same hash ring.

Architecture Refactor and Deployment Optimisation

Replace polling with Kubernetes Informer : A watch‑based informer eliminates control‑plane crashes during large‑scale scaling.

Gradual traffic shift via feature flag : Traffic is ramped from 1 % to 100 % in seconds, shortening deployment pipelines from hours to seconds and enabling instant rollback.

Cost reduction : Skipper pod count dropped from >50 to 8, cutting daily deployment cost from $450 to $110.

Cold‑start warm‑up (N‑Ring fade‑in) : For newly scaled pods, traffic is introduced over 30 seconds following a t^{2.5} curve, warming only the product cache needed for upcoming requests.

Reliability and Fault Bypass

Cross‑AZ‑aware routing was abandoned after experiments showed cache fragmentation and massive DynamoDB read spikes. The final design includes:

Jittered retry for transient failures.

FIFO overload discard policy to protect the system under load.

Node‑level fault detection : Rich client logs identify short‑lived “node freeze” events and automatically bypass the affected pod.

Industry Reaction

Werner Vogels (CTO, Amazon) : “I’m generally skeptical of complex client‑side routing, but Zalando’s implementation is impressive and their lessons are valuable.” Alexey Kuznetsov (Principal Engineer, AWS) : “Seeing client‑side load balancing in action was eye‑opening; this design philosophy deserves more attention.”

Skipper remains an open‑source project on GitHub: https://github.com/zalando/skipper. The in‑process client‑side load balancer is an internal, closed‑source component.