nginx 1.31 least_time Load Balancing: 9x Latency Drop, Streaming Pitfalls & EMA Internals

Background: Eleven-Year Wait for Open Source

nginx 1.31.0 (released 2026-05-13) open-sourced the least_time load-balancing method, previously exclusive to NGINX Plus since 2015. The feature arrived via PR #1306 for both HTTP and stream modules. Official documentation notes it appeared in version 1.7.10, but that refers to the Plus lineage; the open-source mainline only received it in 1.31.0.

Relation to Existing Algorithms

round-robin (default): weighted round-robin, ignores latency. least_conn (since 1.3.1): selects server with fewest active connections (weighted). Indirectly reflects speed but measures connection count, not latency itself. least_time (since 1.31.0 OSS): selects server with lowest average response time + fewest active connections (weighted). random [two] (since 1.15.0): picks N random servers then applies least_conn or least_time among them.

Syntax and Semantics

Syntax:  least_time header | last_byte [inflight];
Default: —
Context: upstream

Requests are sent to the server with the shortest average response time and fewest active connections (weighted); ties resolved by weighted round-robin. header: uses time to first byte (TTFB, $upstream_header_time) as response time. last_byte: uses full response time ( $upstream_response_time). inflight (optional): includes in-flight (uncompleted) requests in the calculation.

The distinction matters when first byte and last byte are far apart — large file downloads, SSE, streaming APIs. header prefers fastest start; last_byte prefers fastest completion.

Source Code Mechanism

3.1 Average Time Update: EMA with α = 0.05

Each peer maintains header_time, response_time, inflight_time updated via exponential moving average:

/* exponential moving average + rounding */
#define ngx_http_upstream_response_time_avg(avg, v) \
    *(avg) = (*(avg) ? (0.5 + ((double) (v) * 0.05 + (double) (*(avg)) * 0.95)) \
              : (v))

New sample weight 5%, history 95% (rounded). Only successful attempts update the average; failed/retried attempts are ignored to avoid biasing toward failing peers. When header ... inflight is configured, a peer.notify callback updates header_time immediately upon receiving response headers — an event-bus pattern inside nginx.

3.2 Server Scoring: Estimated Completion Time = Avg Response Time × (1 + Active Connections)

The function ngx_http_upstream_least_time_eta() computes a score per candidate server:

switch (mode) {
case HEADER:   rt = peer->header_time;  break;
default:       rt = peer->response_time; // last_byte
}
if (now - peer->checked > peer->fail_timeout) {
    /* time decay: halve stored avg each fail_timeout period */
    rt >>= (now - peer->checked) / (peer->fail_timeout + 1);
}
if (peer->inflight_reqs > 0) {
    rt = ngx_max(rt, peer->inflight_time);
}
if (rt > 5000) {
    rt = 5000; // cap at 5s
} else {
    rt += 20 - rt % 20; // round up to 20ms multiple
}
return rt * (1 + peer->conns);

Four-step process:

Base value : EMA average (header or last_byte).

Three corrections :

Time decay: if a server hasn't been selected for a fail_timeout period, its stored average halves each period, allowing recovery.

5-second cap: averages above 5000ms are treated as 5000ms; beyond that, differentiation degrades to least_conn.

20ms rounding: averages rounded up to nearest 20ms multiple. Servers in same bucket are tied and load-balanced via weighted round-robin, preventing over-concentration on millisecond differences.

Multiply by (1 + active connections) : new request queues behind existing ones; each active connection adds one average response time to estimated wait.

Divide by weight : comparison uses cross-multiplication to avoid division; higher weight reduces effective score.

3.3 Summary

least_time

maintains smoothed response-time stats (EMA, time decay, 20ms bucketing), estimates completion time as avg × (1 + conns) / weight, picks the minimum; ties resolved by weighted round-robin.

Experimental Setup

Topology

client (WSL curl)
  │ :8080
  ▼
nginx 1.31.5 (LB_METHOD env var switches algorithm)
  │ upstream `backend` (zone 64k shared stats, keepalive 64)
  ├── b1 :8080  JSON≈10ms   /stream: TTFB=5ms   total≈605ms (fast first byte, slow full)
  ├── b2 :8080  JSON≈100ms  /stream: TTFB=150ms total≈160ms (slow first byte, fast full)
  └── b3 :8080  JSON≈300ms  /stream: TTFB=80ms  total≈230ms (middle)
     (host 9001/9002/9003 direct to backends for runtime tuning)

Rust Backend Design (axum)

GET /

: sleep(delay+jitter) then return JSON once — TTFB ≈ total (regular API). GET /stream: send headers immediately, then stream chunks every chunk_ms — TTFB and total decoupled (streaming). GET /set?delay_ms=500: hot-reload delay parameter for adaptivity tests.

Backends configured so the fastest in header mode (b1) is slowest in

last_byte</sub> mode, guaranteeing visible allocation flips.</p><h3>nginx Configuration</h3><p>Template <code>default.conf.template

rendered via envsubst from LB_METHOD env var. Key config:

upstream backend {
    ${LB_METHOD}          # e.g. "least_time header;"
    zone backend 64k;
    server b1:8080 max_fails=3 fail_timeout=3s;
    server b2:8080 max_fails=3 fail_timeout=3s;
    server b3:8080 max_fails=3 fail_timeout=3s;
    keepalive 64;
}

Log format captures $upstream_connect_time, $upstream_header_time, $upstream_response_time. Switching algorithms:

LB_METHOD='least_time header;' docker compose up -d --no-deps nginx

.

Experimental Results

S1/S2 Baseline: round-robin & least_conn

90 sequential + 120 concurrent ×12 requests:

round-robin : equal distribution (30 each), client avg latency 151ms (sequential).

least_conn sequential : same as round-robin (all connections 0 at selection time).

least_conn concurrent : b1 gets 95, b2 17, b3 8 — fast backend releases connections faster, naturally receives more traffic. But it only senses connection count, not latency directly.

S3/S4 least_time: JSON Scenario

Both least_time header and last_byte send all requests to b1 (10ms). Client avg latency drops from 151ms to 16ms (9.4× improvement). Under concurrency, b1's (1+conns) factor grows, shifting a few requests to b2/b3 — automatic balance between utilizing fastest server and queueing delay.

S5 header vs last_byte: Streaming (Most Dramatic Difference)

40 sequential requests to /stream: least_time header: picks b1 (TTFB 5ms) for 36 requests → client avg latency 612ms . least_time last_byte: picks b2 (total 160ms) for 37 requests → client avg latency 196ms .

Same cluster, same backends, one config line difference → 3.1× latency gap. Logs confirm: header mode shows hdr=0.007 rt=0.618; last_byte mode shows hdr=0.152 rt=0.165.

Practice guidance : streaming/large responses where client cares about full response time → last_byte; CDN origin pull where header arrives fast and client pulls slowly → header. Regular buffered APIs: either works.

S6 inflight Parameter: Minimal Difference Under HTTP/1.1

Concurrent 120×12 on /stream: least_time header vs least_time header inflight show nearly identical distribution (b1 73 vs 72). Reason: eta = rt × (1 + conns) already counts in-flight connections (each HTTP/1.1 request uses a dedicated upstream connection). inflight adds value mainly with HTTP/2 multiplexing (one connection carries many requests) or with header inflight combo where notify callback updates stats at header receipt, reacting ~half RTT earlier.

S7 Runtime Adaptivity: Traffic Migrates Fast, Recovers Slow

least_time header

+ JSON backend, no reload/restart:

b1=10ms → 60 requests all to b1.

b1 slowed to 500ms → within 7 requests traffic shifts to b2 (52 requests).

b1 restored to 10ms → after 60 requests only 2 return to b1; b2 still gets 57.

Asymmetry stems from EMA formula: new 500ms samples (5% weight) quickly raise b1's average above b2; but once b1 stops receiving requests, EMA stops updating — recovery relies solely on time decay (halving per fail_timeout =3s). From 500ms to ~62ms takes ~10s (3-4 decay periods), while test ran only ~6s. This is a deliberate trade-off: anti-flapping vs. recovery speed. Production can lower fail_timeout for faster cutback.

Cold start: empty stats → all servers same score (same 20ms bucket) → first few requests round-robin to gather samples. New instances briefly favored until stats reflect true latency; slow_start or warm-up traffic mitigates.

S8 Negative Verification: random's least_time= Parameter Still Plus-Only

Documentation describes random two least_time=header but OSS build (1.31.5) rejects with invalid parameter "least_time=header". Not marked as Plus-only in docs; only the upstream-block least_time directive is open-sourced.

Usage Scenarios & Selection Advice

Good fits for least_time :

Heterogeneous backends (mixed hardware, spot instances) — auto-distributes by measured latency.

Noisy microservice neighbors — EMA + 20ms buckets resist jitter while tracking trends.

Tail-latency mitigation — automatically reduces load on slow instances.

Streaming workloads — header / last_byte semantic split is unique capability.

Caveats:

Recovery inertia (§S7): fast migration away, slow return — anti-flapping design; tune fail_timeout if faster recovery needed.

Stats driven by real traffic: load tests skew stats; canary traffic may not represent full load.

Requires continuous traffic to keep stats fresh; time decay only time-driven correction.

Recommended companions: zone (shared stats across workers), keepalive (accurate RTT without connect overhead), max_fails / fail_timeout (failure exclusion complements latency stats).

Version gate: feature entered mainline 1.31.x (2026-05); verify your stable branch includes it.

Choosing vs least_conn : Large, stable latency gaps in heterogeneous cluster → least_time wins; near-identical latency in homogeneous cluster → 20ms rounding makes it behave like weighted round-robin, least_conn simpler.

Practical Experience Highlights

Direct observability : logging $upstream_header_time / $upstream_response_time makes every routing decision auditable.

Lower ops burden : traffic auto-routes around slow instances; combined with max_fails in error_log, isolation becomes automatic.

Bucket granularity hits sweet spot : 20ms buckets + weighted round-robin on ties balances "performance optimal" and "load balanced" — no single-server overload observed in tests.