How Netflix’s Cloud Gateway Cuts Errors with Adaptive Load Balancing

Goal

Netflix’s cloud‑gateway team aims to reduce errors, increase availability, and improve fault‑recovery because even a tiny error rate impacts millions of requests per second.

Background

Zuul originally used a round‑robin Ribbon load balancer with black‑listing for high‑failure servers. Recent customizations (e.g., sending less traffic to newly launched servers) helped, but some clusters still showed high load‑related error rates, especially when only a subset of servers was overloaded due to cold starts, temporary slowdowns, or hardware issues.

Guiding Principles

Work within existing load‑balancer framework constraints to enable reuse across teams.

Learn from other teams (e.g., choice‑of‑2 and probation algorithms).

Avoid distributed state; prefer local decisions.

Minimize client‑side configuration and manual tuning.

Prefer adaptive mechanisms over static thresholds.

Load‑Balancing Methods

The core idea is to combine client‑side latency data (best obtained from the client) with server‑side utilization data (best obtained from the server) for optimal balancing.

Choice‑of‑2 algorithm for server selection.

Primary balancer using a server‑utilization view.

Secondary balancer using a server‑view based on utilization.

Probation and server‑generation mechanisms to avoid overloading newly started servers.

Statistical decay to zero over time.

Join‑the‑Shortest‑Queue (JSQ) combined with server‑reported utilization

JSQ works well for a single balancer but causes herd‑behavior across a balancer cluster. Combining JSQ with choice‑of‑2 mitigates this issue.

Server‑reported utilization provides a complete view for all balancers, avoiding JSQ’s incomplete data problem.

Implementation Options

Active health‑check endpoint polling each server’s current utilization.

Passive tracking of responses to infer utilization.

We chose the passive approach for lower overhead and fresher data.

X-Netflix.server.utilization: <current-utilization>[, target=<target-utilization>]

Filtering

When randomly picking two servers, any server exceeding safe utilization or health thresholds is filtered out. The balancer makes a best‑effort N‑try random selection before falling back to unfiltered servers.

Operational Impact

Request Distribution Gap

Moving away from round‑robin increased load variance between servers, but the choice‑of‑2 algorithm reduced the gap compared to pure JSQ.

Slow Servers Receive Less Traffic

Traffic now skews toward faster servers, affecting red‑black deployments and can hide under‑performing servers from monitoring.

Dynamic Data Updates

Rolling data updates and phased deployments limit blast radius of failures.

Synthetic Load Test Results

New balancer reduced load‑related errors by several orders of magnitude compared to round‑robin.

Average and tail latency improved three‑fold.

Server‑side errors dropped by an order of magnitude.

Impact on Production Traffic

The new balancer efficiently routes traffic to healthy servers, reducing the need for manual intervention during partial or full degradations.

During Incidents

When a service experienced increasing thread blockage, the balancer sent less traffic to the overloaded servers, and auto‑scaling added fresh instances that handled roughly double the RPS, mitigating the incident.

Conclusion

This article shares a practical load‑balancing approach tested at Netflix scale. By combining client latency, server utilization, and probabilistic selection, the new balancer dramatically reduces load‑related errors and improves real‑world traffic distribution, though teams should adapt the techniques to their own constraints.