How Google’s TurboQuant Cuts LLM Memory by 6× and Speeds Up Inference 8×

TurboQuant: Compressing LLM KV Cache to 3‑bit

Google Research introduced TurboQuant, a novel algorithm that tackles the long‑standing memory‑intensive and slow‑response problems of large‑language‑model (LLM) KV (Key‑Value) caches. By quantizing KV entries from 32 bits to just 3 bits, TurboQuant reduces memory consumption six‑fold and accelerates inference eight‑fold on NVIDIA H100 GPUs, while keeping downstream accuracy at 100 %.

Why KV caches are a bottleneck

LLMs encode text, images, and other modalities as high‑dimensional vectors. These vectors are stored in the KV cache for fast reuse during generation. The cache behaves like a high‑speed “digital sketchpad,” but its size grows linearly with model depth and context length, quickly exhausting GPU memory.

TurboQuant’s two‑step approach

Step 1 – High‑quality compression (PolarQuant) : Randomly rotate vectors and represent them in polar coordinates, separating magnitude (radius) from direction (angle). This captures the main information with most bits while discarding redundant Cartesian components.

Step 2 – Error elimination (QJL) : Apply Quantized Johnson‑Lindenstrauss (QJL) to encode the residual error using a single bit per dimension (+1 or –1) and a specialized estimator that restores the original precision.

The combination yields “extreme compression with zero precision loss.”

PolarQuant: From Cartesian to polar representation

Traditional vector quantization stores X, Y, Z coordinates and extra quantization constants, which adds overhead. PolarQuant instead groups coordinate pairs and maps them to a radius and an angle, dramatically reducing the number of stored values. 向东走 3 个街区，向北走 4 个街区 In Cartesian terms this is (3, 4). PolarQuant would store the same point as a radius and angle, e.g. “以 37 度角，走 5 个街区”. 以 37 度角，走 5 个街区 The radius encodes data intensity, while the angle encodes direction or semantic meaning. Because the angular pattern is known and bounded, the model no longer needs costly data‑normalization steps.

QJL: Quantized Johnson‑Lindenstrauss

QJL reduces each vector component to a single binary symbol (+1 or –1). A specially designed estimator then corrects the tiny errors introduced by this extreme quantization, effectively performing a “mathematical health check” on the compressed data.

Experimental results

Memory reduction : KV cache size shrinks by 6×, enabling consumer‑grade GPUs to handle much longer contexts.

Inference speed : On H100 GPUs, TurboQuant delivers an 8× speedup for attention‑logits computation.

Accuracy : In the “Needle‑in‑a‑Haystack” benchmark, TurboQuant matches full‑precision models with perfect downstream results.

Vector search : TurboQuant outperforms PQ, RabbiQ, and other state‑of‑the‑art quantization methods on GloVe retrieval tasks, achieving the highest 1@k recall without large codebooks or dataset‑specific tuning.

Figures from the LongBench and Needle‑in‑a‑Haystack evaluations (using open‑source LLMs such as Gemma and Mistral) confirm these gains.

Implications for AI deployment

Large models become cheaper to run because memory savings lower hardware costs.

AI applications can be deployed on less‑expensive devices, bringing “AI‑on‑the‑edge” closer to reality.

Vector‑search engines become faster and more accurate, benefiting semantic search and retrieval systems.

The underlying algorithms contribute fundamental advances to quantization theory, with provable guarantees.

In short, TurboQuant demonstrates that algorithmic innovation can outpace raw hardware scaling, reshaping the future of AI systems.