FlashStates: 3x Prefill Throughput for Hybrid LLM Agents via Flexible State Caching
FlashStates introduces arbitrary state checkpoints and cost-aware eviction for hybrid LLM architectures, achieving up to 3.14x prefill throughput and 88% TTFT reduction on real agent workloads, with CPU cache integration and cross-engine reuse.
