FlashStates: 3x Prefill Throughput for Hybrid LLM Agents via Flexible State Caching
FlashStates introduces arbitrary state checkpoints and cost-aware eviction for hybrid LLM architectures, achieving up to 3.14x prefill throughput and 88% TTFT reduction on real agent workloads, with CPU cache integration and cross-engine reuse.
Hybrid LLM architectures combining full attention with linear attention or state-space layers (e.g., Kimi Delta Attention, Gated DeltaNet, Mamba-2) are used in models such as Kimi K3.1 Pro, GLM-5.3-Flash, Qwen3.8-Flash-Next, and Nemotron 3 Super. These models achieve faster decoding and smaller KV caches, but prefix caching is more complex because linear layers use a fixed-size recurrent state that cannot be rolled back: to resume from position p, both the full attention KV and the exact recurrent checkpoint at p are required — together called the "hybrid state".
Problem: Three Avoidable Cache Misses in Agent Workloads
Agent workloads consist of multi-turn, long-context requests where the ideal reuse point is the end of the previous turn's decode output, which rarely aligns with engine block boundaries. The authors ran GLM-5.3-Flash on vLLM and SGLang, tracing every recomputed prompt token to identify three miss types:
Placement miss : checkpoint not saved at the needed position. vLLM saves checkpoints only every 4,352 tokens during prefill and none during decode; at low concurrency (8–16) placement misses account for 43% of uncached tokens.
Dependency miss : KV remains but its dependent checkpoint was evicted. SGLang uses finer 256-token blocks, eliminating placement misses (<5%), but ~430 checkpoints crowd a separate state pool; LRU eviction often discards checkpoints whose KV is still resident, causing dependency misses.
Capacity miss : cache simply too small. At higher concurrency (32–64), both engines enter "hybrid state thrashing" where capacity misses compound dependency misses.
Conclusion: large checkpoint intervals cause placement misses, small intervals cause dependency misses, and high concurrency adds capacity misses — tuning density alone cannot solve all three.
FlashStates Design
1. Arbitrary State Checkpoints (ASC)
FlashStates captures the exact terminal state of each request at completion. If the endpoint falls inside a KV block, the entire tail block (both attention KV and recurrent checkpoint) is retained under unified management. For mid-prefill positions, a PrefillExport interface lets kernels export state at specified positions without altering output; KDA and GDN kernels are already adapted.
2. Optimization Objective
With ASC, hybrid state can exist at any position. The goal: under GPU memory budget B, minimize the sum of redundant prefill time across all requests. For a request r, define ideal recovery point ℓ(r)* (longest prefix match with any prior content) and actual recovery point ℓ(r)(π) (latest position where full hybrid state survives under policy π). The cost C(ℓ(r)(π), ℓ(r)*) is the prefill time for tokens between them. The objective is to minimize Σᵣ C(ℓ(r)(π), ℓ(r)*).
3. Dependency-Aware Admission and Eviction
Building on GreedyDual-Size (GDS), each hybrid state gets priority H(u) = last-access watermark + (recovery cost ÷ blocks actually freed on eviction). Recovery cost is the extra prefill time estimated by the cost model. Eviction unit is "one checkpoint + blocks exclusively depended on by it"; shared prefix blocks are not counted as freed. Both terms share units of "prefill time per block", so they add directly without load-dependent hyperparameters. Admission uses the same priority: a new checkpoint is kept only if its priority exceeds the lowest-priority cached item.
GPU Cache Experiments
Experiments on NVIDIA B200 GPUs using SemiAnalysis's public Claude Code traces (GLM-5.3-Flash, 1M context, 48 sessions):
Prefill throughput vs. vLLM: 3.15× (GLM-5.3-Flash), 2.27× (Nemotron 3 Super), 1.54× (Qwen3.8-Flash-Next).
p90 TTFT reduction: 88.4%, 77.1%, 79.0% respectively.
Performance holds across concurrency levels, eliminating the thrashing seen in baseline engines.
Beyond GPU: CPU Cache and Cross-Engine Reuse
Production systems offload excess cache to CPU memory or shared storage. FlashStates integrates with Mooncake (512 GiB CPU cache) via three engineering changes:
Async write-through on request completion : new terminal state is queued for CPU copy immediately, not waiting for GPU eviction.
Two-level unified management : Mooncake's metadata server tracks state ownership and shared KV references, extending dependency-aware GDS to GPU+CPU tiers so blocks still depended on are never mistakenly deleted.
Transfer optimization : optimized GPU→CPU copy and compression of speculative decoding (MTP) related state.
Tests with GLM-5.3-Flash NVFP4, 1M context, 2×B200 per server, 3-step MTP, comparing against vLLM+Mooncake and SGLang+HiCache (511.5 GiB host cache):
Output throughput up to 41.4% higher than vLLM+Mooncake, 36.2% higher than SGLang+HiCache.
Median uncached prompt tokens reduced 68.7%–81.4%.
At 80 sessions, p90 TTFT ~7.18 s vs. 21.88 s (vLLM) and 22.97 s (SGLang) — roughly 3× improvement.
Furthermore, states written to shared storage can be reused by another engine worker on a subsequent turn, provided the prefix matches exactly and all tensor-parallel (TP) rank objects are present. This enables prefill-decode (PD) disaggregation where the next turn's prefill runs on a different engine.
Conclusion and Adoption
FlashStates' core principle: recovery position should be determined by "where reuse is possible", not by "how KV is blocked". It is integrated into vLLM, covers three hybrid architectures, supports CPU cache and speculative decoding, and prepares for cross-engine state sharing. As hybrid architectures spread and agent contexts grow, inference systems designed for pure Transformers need hybrid-state-aware cache redesign. Notably, SGLang has already adopted FlashStates' cost-aware state eviction policy, documented in its official radix eviction policy (https://docs.sglang.io/docs/advanced_features/radix_eviction_policy#state_cost-parameters).
References
Paper: FlashStates: Unlocking Flexible Hybrid State Management for Agentic LLM Inference (https://www.alphaxiv.org/abs/2611.FlashState)
Code: https://github.com/FutureMLS-Lab/FlashStates
Project homepage: https://FlashStates.ai
Signed-in readers can open the original source through BestHub's protected redirect.
This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactand we will review it promptly.
How this landed with the community
Was this worth your time?
0 Comments
Thoughtful readers leave field notes, pushback, and hard-won operational detail here.
