Why Most Tokens Are Needlessly Recomputed and How LMCache Goes Beyond KV‑Cache

Stanford researchers measured that 62% of the payload in each AI‑Agent call is duplicated (unchanged system prompts, tool definitions, and knowledge‑base documents), yet the model recomputes the entire KV cache for every request.

Prefix (prompt) caching can reduce input cost by up to 90% for fixed prefixes and achieve hit rates of 60%–85% in stable scenarios, but it requires the cached content to be an exact prefix, making it fragile.

RAG multi‑document retrieval – caching documents A and B separately fails when a request needs both, because B was cached without A preceding it.

Document order changes – rearranging the same set of documents invalidates all cached blocks.

Multi‑turn dialogue – each new turn shifts token positions after the stable prefix, invalidating downstream cache.

The root cause is that KV cache is position‑dependent: each token’s KV pair encodes attention to all preceding tokens, so a cache block is only valid in the exact context where it was computed.

Alibaba Cloud production data shows that 10% of cache blocks contribute 77% of hits, while the remaining 90% occupy space without ever being used.

Existing in‑process cache architectures also slow inference because KV‑cache read/write, data movement, and matrix multiplication compete for the same GPU resources.

Google’s TurboQuant compresses KV cache to 3 bits without precision loss, but running the compression in the same inference process adds over 20% latency, demonstrating that compression alone does not resolve resource contention.

LMCache separates cache management into an independent process. The inference engine exchanges block IDs via shared GPU memory, while the LMCache process moves large tensors in parallel, analogous to a chef delegating ingredient fetching to a dedicated helper.

No resource contention – the 20% inference slowdown caused by compression and movement disappears.

Zero‑copy multi‑GPU sharing – multiple GPUs can read/write the same memory region without repeated copies.

Hierarchical parallel lookup – caches can reside on GPU, CPU, SSD, or remote storage; LMCache queries all layers simultaneously and fetches from whichever layer holds the data.

Although the separate‑process design solves performance bottlenecks, the inherent limitation of prefix caching remains. LMCache’s CacheBlend technique (EuroSys 2025 Best Paper) observes that most tokens attend only to nearby context, while only a small fraction require cross‑document attention. CacheBlend recomputes those cross‑boundary tokens and reuses the remaining cached KV pairs, regardless of document order or composition.

For RAG pipelines this means each knowledge‑base document becomes a reusable cache asset. Real‑world tests report a 2×–4× speedup for multi‑document processing with no quality loss.

Benchmark on an H200 GPU running Qwen3‑235B with 50 concurrent users shows that LMCache reduces first‑token latency to 1/14 of traditional in‑process caching, improves decode speed by 4×, and cuts startup time from over 3 minutes to about 30 seconds.

LMCache is production‑ready: it includes Prometheus and OpenTelemetry monitoring, Kubernetes deployment, CLI debugging and load‑testing tools, and fault tolerance (cache survives inference crashes; inference degrades gracefully if LMCache fails). It supports major inference engines (vLLM, SGLang, TensorRT‑LLM) on NVIDIA and AMD GPUs, and storage back‑ends such as CPU memory, local SSD, Redis/Valkey, Mooncake, InfiniStore, S3‑compatible object stores, NIXL, and GDS. It also offers PD‑separation (prefill/decode decoupling) and a pluggable KV‑transform interface for research on compression, token dropping, or custom serialization.

Adoption metrics: >10 k GitHub stars, acceptance by the PyTorch Foundation, integration into NVIDIA Dynamo, and Apache 2.0 licensing. The project epistemic‑graph uses LMCache as its default L2 KV‑cache persistence layer.

Token pricing fell from $30 per million tokens in 2023 to $0.40 in 2026 (an 80% reduction), but agent workloads consume 5–30× more tokens per task because each step resends the full context. Uber’s Claude Code deployment exhausted a year’s budget in four months; Gartner predicts 40% of AI‑Agent projects will be cancelled by 2027 due to cost overruns.

A single MI300X GPU can generate roughly 15 TB of KV cache per day, most of which is discarded.

Project repository: https://github.com/LMCache/LMCache