How KV Cache and CachedAttention Revolutionize LLM Inference Efficiency

Background

As large‑language‑model (LLM) training scales up, the inference stage becomes the dominant cost driver, especially for multi‑turn conversations where repeated attention calculations consume valuable GPU high‑bandwidth memory (HBM) and increase latency.

KV Cache Mechanism

KV Cache stores the Key and Value vectors generated for each token during the prefilling phase, allowing the model to reuse these vectors in subsequent decoding steps without recomputing them, thereby cutting redundant work and speeding up token generation.

Challenges with HBM and SSD

HBM offers excellent bandwidth but is expensive and limited in capacity; as dialogue length grows, KV Cache size can quickly exhaust HBM and even DRAM, forcing costly data movement and repeated calculations.

CachedAttention: Storing‑instead‑Computing

CachedAttention introduces an external low‑cost storage medium (AttentionStore) that holds KV Cache data on a combination of HBM, DRAM, and SSD. When a conversation becomes inactive, its KV Cache is moved to the store instead of being discarded, and it is reloaded on demand for later turns, eliminating repeated computation of historical tokens.

Multi‑Level KV Cache Architecture

HBM (GPU local high‑bandwidth memory) – fast tier for active session KV Cache, directly feeding the attention and feed‑forward networks.

DRAM (host memory) – intermediate buffer that holds recently accessed inactive KV Cache and serves as a write‑back target for HBM overflow.

SSD (persistent storage) – large‑capacity tier that stores long‑term KV Cache pools, preventing loss of rarely accessed sessions.

Performance Results

Experimental evaluations show that CachedAttention reduces first‑token latency (TTFT) by up to 87%, improves prefilling throughput by 7.8×, and lowers end‑to‑end inference cost by roughly 70%.