Why KV Caching Is Critical for Efficient LLM Inference

Idea 1: Q/K/V Dynamics Differ Between Training and Inference

In inference the Q/K/V weight matrices remain fixed, whereas during training the actual Q/K/V values change after each batch. Consistent inference results require a fixed random seed, identical hardware/software floating‑point handling, and deterministic decoding settings such as temperature = 0 and a fixed Top‑K.

Because the same input token yields identical outputs under these conditions, any intermediate computation that repeats can be avoided by caching.

Idea 2: Prompt Design Drives KV Caching Benefits

Prompts are a major innovation in LLMs, heavily influencing output quality. The "Prefix‑Decoder" architecture (e.g., Tsinghua KEG Lab’s GLM series) computes K and V for the prompt once during the Prefill stage, then reuses them during the Decoder stage. Most LLMs, such as LLaMA and Qwen, still use a causal decoder, where Prefill‑Stage KV caching is called Prefix‑Caching.

KV caching in the Prefill stage reduces Time‑to‑First‑Token (TTFT), while caching during decoding lowers Generation Time (GT) and inter‑token latency (ITL).

Idea 3: Bigger Cache Is Not Always Better

KV caching targets repeated tokenizations. Caching every token would consume excessive memory; unused entries waste resources. An oversized cache can even increase per‑token lookup time. Therefore, caching is only worthwhile when a token’s short‑term reuse count exceeds one.

The goal is to cover short‑term repeated KV values with minimal memory, keeping query time low.

Idea 4: KV Caching in Batch‑Concurrent Inference

When serving many requests concurrently, KV caches may be distributed across nodes. The planning of Prefill and Decode stages becomes more complex, with different concurrency strategies for short‑term batch processing.

Idea 5: Minimizing Per‑Request KV Cache Footprint

Research aims to reduce the memory each request occupies in the KV cache. Typically, KV cache consumes about 30 % of total memory; lower fragmentation allows more requests to fit, improving utilization.

Reducing fragmentation within this 30 % slice lets more requests share the cache, raising overall efficiency.

Conclusion

Understanding KV‑Cache mechanics is straightforward, but practical deployment faces challenges such as memory budgeting, prompt design, and concurrent request handling. State‑of‑the‑art systems like FlashAttention, Paged Attention, RadixAttention, vLLM, and SGLang implement sophisticated KV‑caching strategies to balance latency and throughput.

References:

https://medium.com/@joaolages/kv-caching-explained-276520203249

https://developer.nvidia.com/blog/mastering-llm-techniques-inference-optimization/

https://docs.nvidia.com/nim/benchmarking/llm/latest/metrics.html

https://huggingface.co/blog/tngtech/llm-performance-prefill-decode-concurrent-requests