Stabilizing LLM Inference: Traffic Shaping, Dynamic Load Balancing & KVCache Awareness

Background: Growing Context Length and Session Turns

With the rapid development of Agentic AI, inference services face two major shifts: average context length grew from a few thousand tokens in 2024 to 60K–90K in 2025–2026, and session turns increased from ~1.5 to dozens or hundreds. These changes raise KVCache hit rates, which act as both a performance lever and a risk lever — higher hit rates yield greater gains, but sudden traffic pattern changes causing hit-rate drops amplify capacity pressure.

A concrete example: if cache hit rate falls from 90% to 40%, prefill compute share jumps from 10% to 60%, GPU demand may increase 3–5× at the same QPS. Combined with load imbalance, decode throughput can drop over 30%, triggering queue timeouts and upstream retries that further worsen overload.

Overall Architecture

The inference service stack (built by Baidu Baige and Qianfan) comprises five layers: Gateway Access, Traffic Scheduling, Load Scheduling, Prefill/Decode Resource Pools, and KV Storage Foundation. The stability work targets three layers converging bottom-up: first stabilize traffic, then balance load, finally protect cache hit rate and capacity safety.

1. Traffic Scheduling & Elasticity

Challenges with Traditional Approaches

Provisioning for peak: high cost, hard to predict amplification from pattern shifts.

Uniform rate limiting: indiscriminately rejects high-value long sessions and short requests, causing cliff-edge availability drops.

Reactive scaling: large model weights, pin-memory registration, and image preparation cause cold-start latency (minutes), often too slow to catch flash crowds.

Solution: Traffic Shaping + Fast Capacity Follow-up

Multi-level Traffic Control & SLO-aware Scheduling

Traffic is classified into three priorities (high/medium/low) based on input length and cache hit rate. Under capacity pressure, quotas are tightened progressively (tiered rate limiting) rather than a single cutoff. At the scheduling entrance, priorities determine queue weights and admission quotas; high-priority traffic enters the engine first.

Inside the engine, the scheduler makes dynamic SLO decisions: for long or low-hit-rate requests, it adjusts wait budgets using upstream remaining SLO to reduce unnecessary timeouts and recomputation; requests predicted to exceed SLO even after queuing are rejected early to avoid pile-up; requests nearing SLO expiry are prioritized to improve overall SLO attainment.

Cold-start Acceleration

Two bottlenecks were addressed:

Pin Memory registration : previously each GPU context required serial page-pinning and host-memory registration, taking >5 minutes at scale. Switched to shared GPU memory via CUDA IPC — only the first process performs registration; subsequent processes just map handles, reducing this step to near zero.

Image preparation in mixed-model pools : frequent cache misses during multi-model deployment caused unpredictable startup. Pre-staging images cut load time by ~70%.

Combined, engine deployment time dropped ~60%.

Event-driven Minute-level HPA

An Event Plane collects per-second engine TPS, scale-out TPS, batch utilization, and block utilization. It forecasts traffic trends to pre-reserve capacity and back-calculates required headroom from scale-out latency. HPA triggers scale-out on any key metric breach; scale-in occurs only when all metrics fall below target. Anti-thrashing guards include per-step scale-in limits, cooldown periods, and min/max instance bounds. The loop runs at minute granularity, feeding results back into the Event Plane.

Results

Customer-perceived availability jitter incidents reduced 80%.

Pin-memory registration latency → near zero.

Image load time ↓ ~70%.

Capacity water-level awareness and scaling decisions at minute level.

2. Uncontrollable Load: Dynamic Bucketing & Cross-bucket Hot Migration

Three Types of Imbalance

Intra-batch imbalance : long and short requests in same batch; short requests wait for long ones, dragging batch throughput.

Instance idling in EP/DP parallelism : slow data-parallel ranks hold back fast ones, leaving completed instances spinning.

Dynamic drift : as decode progresses, context length crosses static bucket boundaries, invalidating the original assignment and causing performance regression.

Dynamic Bucketing (Static Snapshot)

Define per-token capacity cost : benchmark fixed hardware across request lengths to get capacity ceilings, then convert to per-token consumption ratio.

Online load distribution : scan request-length axis in 500-token atomic buckets, aggregate load per bucket.

Baseline target : total cluster load / instance count = target load per instance (L_target = L_total / N_instance).

Scan & split : accumulate atomic buckets until reaching L_target, assign corresponding instance count, repeat to cover full length axis, output new bucket-to-instance mapping and rebind instances.

Recomputation runs every ~30 seconds, matching instance resources to real-time load distribution.

L_target = L_total / N_instance

Cross-bucket Hot Migration (Runtime Drift)

When a request's length crosses its bucket boundary, a three-phase state machine migrates its KVCache without interrupting token output:

Prepare : Rescheduler picks target decode instance. Source creates checkpoint, continues emitting tokens while asynchronously shipping existing KV; target pre-allocates GPU memory and validates capacity/topology.

Freeze/Sync : after bulk transfer finishes, rescheduler signals switch. Source pauses inference briefly, syncs only incremental KV and latest token; target verifies integrity and takes over subsequent batches. Pause window is minimal because only delta is synced.

Commit : rescheduler confirms handoff, token stream redirects to target. Source releases KV and metadata. On any failure, rollback to source continues generation, guaranteeing uninterrupted user output.

Effectiveness

Softmax attention models: decode throughput +40–50%, prefill throughput +20–30%.

Bucket recomputation cycle ~30 seconds.

Linear/Hybrid attention models show less benefit; enable per-model-family rather than globally.

3. Cluster Cache Absolute Stability

Hit-rate Instability Sources

Traffic spikes & larger models → faster KV generation, larger per-token KV footprint. Insufficient storage triggers aggressive eviction, ejecting even high-hit-rate historical KV.

New traffic patterns (e.g., influx of new users with low hit-rate, few-turn requests) structurally depress overall hit rate; cannot be fixed by scaling alone.

Multi-level KV Storage Evolution

2024 : Single-node L2 memory cache — hundreds of GB, ~20 GB/s bandwidth. Sufficient for short contexts, few turns.

2025 : Added L2.5 SSD cache — 4.5 TB/node, 10–20 GB/s. Capacity jumped an order of magnitude; bandwidth not a bottleneck.

2026 Q1 : L2 + L2.5 P2P pooling with UVA — bandwidth 30–40 GB/s, broke single-node capacity boundary, hit rate significantly improved.

Next : L3 distributed storage — decouple capacity & bandwidth, scale independently, push actual hit rate closer to theoretical limit.

Global Cache Awareness for Cross-region Scheduling

Previously, scheduling only considered compute load per region, ignoring session KV distribution. Long sessions could be routed to regions lacking their history KV, causing redundant prefill and secondary overload.

Now the gateway aggregates per-region cache distribution and instance load into a global cache view, feeding scheduling with two signals: hit probability + load water-level.

Miss requests: prefer lower-load regions to balance globally.

Hit requests: retain in current region, especially those with high hit-round counts and long established prefixes, to avoid repeat prefill.

Result: more even, stable cross-region request distribution; reduced chain reactions from single-region overload.

Full-link Hit-rate Funnel for Observability

Decomposes hit rate into four layers to pinpoint where drops occur:

Theoretical ceiling : infinite KV space, no eviction.

Entrance expected hit rate : after finite space, LRU eviction, L2/L2.5/L3 tiering policies.

Post-scheduling expected hit rate : losses from cross-region scheduling, cross-instance migration, global cache view effectiveness.

Engine actual hit rate : engine-side eviction, load hotspots, tiered storage read-speed convergence.

Quantifying inter-layer gaps turns "hit rate dropped" into "which layer dropped by how much", enabling online prevention, rapid root-cause, and timely mitigation.

Summary

Traffic side : tiered control, cold-start acceleration, event-driven elasticity → 80% fewer availability jitters, drastically shorter scale-out prep.

Load side : dynamic bucketing + KVCache hot migration → decode throughput +40–50%, prefill +20–30%.

Cache side : global cache scheduling, multi-level storage, full-link funnel → hit rate stabilized >90% of theoretical max, cross-region load significantly flattened.

All built on SGLang; open to community collaboration. Core message: inference stability isn't about piling resources, but precise perception and control of load, traffic, and cache.