Baidu Intelligent Cloud Tech Hub

Oct 18, 2026 · Artificial Intelligence

Stabilizing LLM Inference: Traffic Shaping, Dynamic Load Balancing & KVCache Awareness

Baidu Baige presents a three-layer stability framework for LLM inference services: tiered traffic control with minute-level autoscaling, dynamic request bucketing with cross-bucket KVCache migration boosting decode throughput 40-50%, and multi-level KV storage with global cache awareness keeping hit rates above 90% of theoretical maximum.