Baidu's KVCache Overhaul: 36% TTFT Cut via Hit-Rate & Multi-Chip Tuning

Cache Hit Rate Optimization

Three-Layer Cache-Aware Scheduling

Baidu's Qianfan platform uses a three-layer scheduling architecture: global gateway → regional cache-aware scheduler → cluster-internal cache-aware scheduler. The regional scheduler ensures session affinity so multi-turn requests from the same session hit the same cluster, reusing existing KV Cache. The cluster-internal scheduler jointly optimizes cache hit rate and load balancing. Under low load, requests are steered toward nodes with the highest cache hit rate; under high load, a pooling mechanism migrates cache to the request's node to maintain cluster-wide load balance, compensating for hit-rate loss caused by load balancing.

Decoupled Cache System with Event-Driven Global Index

The cache system runs as independent processes (Agent per node + regional Master), decoupling cache lifecycle from the inference engine. Data plane and control plane are separated; an event-driven asynchronous mechanism builds a global index. The scheduler subscribes to incremental KV events and performs local retrieval, enabling timely awareness of global cache distribution.

Prefetch with Interruptible Preload

To reduce pooling overhead, the system implements prefetch and interruptible prefetch. When the scheduler assigns a request to a node that is not the highest-hit-rate node, it notifies the local Agent to prefetch cache from the high-hit-rate node into local L2 (host memory). As soon as the engine starts loading cache into L1 (GPU memory), prefetch stops immediately, avoiding interference with engine execution. Prefetch covers both local (SSD → host memory) and remote (RDMA from peer node's host memory).

Scenario-Specific Optimizations

Dynamic SWA save stride: For Sliding Window Attention (SWA), which only hits at saved positions, the save frequency is increased in shorter length intervals to improve system prompt hit rate.

PP parallel cache management: In pipeline parallelism, external cache is shared across ranks, but match lengths can diverge while GPU requires uniform shapes. Traditional engine-side alignment (as in SGLang) conflicts with external cache because PP stages cannot synchronize directly. The solution makes the cache system PP-stage-aware: when stage 0 matches, the system locks the hit cache for subsequent stages; only after all stages finish loading is the cache released, achieving synchronization outside the engine and ensuring consistent read lengths across stages.

Cache Performance Overhead Reduction

Overhead Metric

Cache overhead is defined as:

Cache Overhead = 1 - (Prefill throughput on non-hit tokens with cache enabled) / (Prefill throughput with cache disabled)

With cache disabled, prefill efficiency is maximal. Enabling cache raises total throughput but reduces non-hit token processing efficiency; the drop is the cache overhead.

Overhead Breakdown

Overhead splits into compute overhead (natural compute increase from longer context) and gap overhead (idle gaps between GPU computations due to cache management). Gap overhead accounts for >80% of total, dominated by API copies and Radix Tree management.

Economic Constraint

At 80% hit rate with a 10× discount on hit tokens, cache overhead must stay below 28.6% to cover the revenue gap from discounting.

Optimization Techniques

Rolling Hash offload: Rolling Hash for L2 indexing was moved from the engine to the tokenizer, eliminating GPU interference.

Radix Tree optimization: Adopted more efficient Bigram key representation and indexing, and re-implemented Radix Tree in C++, drastically cutting index management cost.

UVA copy: L2 uses Page-First layout while L1 uses Layer-First, leading to fine-grained copies. Unified Virtual Addressing (UVA) copies reduce kernel launch overhead and improve small-copy performance.

MLA-class cache optimization: For MLA (Multi-Head Latent Attention) and similar (DeepSeek V4 CSA/HCA), each rank holds identical latent vectors. Each GPU reads only 1/8 of data via PCIe, then aggregates via NVLink AllGather. H2D copy and NVLink scatter run in parallel per layer, further reducing read/write overhead.

Multi-Chip Adaptation for Kunlun XPU (HiCache L2)

L2 Layout Choice: Layer-First vs Page-First Direct

L1 (GPU memory) is fixed Layer-First (contiguous per layer). L3 (storage) requires Page-First (contiguous per page) for large sequential I/O and zero-copy. L2 sits in between. Layer-First matches device side, enabling direct layer-wise copy and merge in page direction, but fragments a page into dozens of segments, degrading L3 to many small I/Os and losing zero-copy (mooncake backend unsupported). Page-First Direct gives contiguous pages for L3 single large I/O, but mismatches device layout, requiring layout transformation on every transfer.

If only L2 exists, Layer-First is preferable; if L3 is needed, Page-First is mandatory.

DMA Segment Explosion & Batched Gather/Scatter

With Page-First Direct default implementation, host-side same-layer different-page segments are separated by (layers × page_size), never mergeable; device-side each layer is a separate buffer, blocking layer-wise merge. Combined, segment count = pages × layers. Example: 32K tokens, page_size=64, 78 layers → 39,000+ DMA segments. Layer-First ideal merge yields only 78 segments — two orders of magnitude fewer. The bottleneck is CPU-side DMA descriptor submission, not PCIe bandwidth.

Solution: batch transfers. On write-back, gather all layers into a contiguous page; on load, scatter back to layered layout. Layout unchanged, L3 retains zero-copy, submission count drastically reduced. Page-First Direct performance matches Layer-First, and scatter supports per-layer chunking to overlap with forward pass.

Bidirectional Async Transfer

D2H and H2D are asynchronous but submission occurs on the scheduler main thread; large transfers block the scheduling loop. Implemented dedicated Backup and Load threads submitting on private streams; main thread only posts tasks and consumes completion events. In testing, bidirectional async reduced E2E latency by 14% vs synchronous submission.

Host Memory & Link Optimization

Pinned huge pages: Host KV pool default 4KB anonymous pages create massive page tables. Applied madvise to collapse into 2MB huge pages, then HostRegister. Mapping entries drop ~500×, benefiting CPU page tables, IOMMU, and device DMA translation paths simultaneously.

NUMA affinity: Memory registration competes for local memory and PCIe lanes. Without binding, HiCache 80GB degrades 20.6%, 40GB degrades 11.5%, 15GB degrades 8.8% — degradation correlates with registered memory size, not transfer volume. Binding to XPU topology eliminates degradation and makes it insensitive to HiCache size.

PCIe relaxed ordering: On some platforms, strict NUMA binding forces two XPUs to share an upstream link. Their D2H writes have different source IDs and target NUMA nodes but strict ordering makes them wait on the shared upstream. Switching to ID-based relaxed ordering restores D2H bandwidth to 4× the strict-ordering baseline.

Framework-Level Optimizations

write_through_selective with Ghost List: Original policy writes only reused prefixes (hit count > 0). Under L1 pressure (long context, high concurrency), prefixes evict before second access, count resets to zero, true reuse never confirmed. Added Ghost List: on L1 eviction, drop KV but keep 64-bit page hash (configurable capacity, O(1) insert/evict, LRU when full). On prefix rebuild, a Ghost List hit proves reuse, so write directly to L2. This extends the "reuse recognition window" from L1 residency time to Ghost List memory window, which is orders of magnitude larger, making selective write-through practical.

LIFO slot allocation: Idle slot allocator changed from FIFO (full-pool scan, scattered writes) to LIFO (reuse just-freed slots). Writes concentrate on a small hot page group. No impact on eviction semantics or hit rate; purely a memory locality optimization.

TP-Group Shared Host KV Buffer

MLA / DeepSeek V4 CSA / HCA store identical latent vectors per TP rank. With TP=8, L2 holds 8 bitwise-identical replicas → effective capacity 1/8 nominal, 8× D2H traffic. DP Attention mitigates but redundancy persists while TP≠1.

Solution: one shared host memory buffer per TP group. All ranks map same physical memory; rank 0 writes, others read-only. Non-rank-0 still report completion events, so node readability still gated by rank 0 write completion — no new cross-rank sync introduced.

Result: same memory, TP=8 yields 8× effective capacity, 8× fewer D2H submissions and volume, 8.3% further TTFT reduction. Benefits cascade to L3: preload volume to L2 drops 8×, L3→L2 sync also improves.

Shared Buffer Placement: Interleaved vs Socket-Local

Two placement policies: cross-NUMA interleaving (capacity-first, spreads pool across all NUMA nodes, avoids hotspots) vs per-socket grouping (bandwidth/latency-first, avoids cross-socket access, sacrifices capacity by socket count factor).

End-to-End Results

PD separation, real Agent dataset, HiCache L3 disabled, no composite routing. Baseline = HiCache disabled:

Initial adaptation: TTFT -23.4%

+ Host memory optimizations: -27.7%

+ TP-group sharing: -33.7%

+ L2 capacity converged to hit-rate saturation point: -36%

Future Work

Focus shifting to L3, L3.5, and L4 layers, exploring storage pooling and dedicated storage hardware for Agent inference scenarios.