How Google’s TurboQuant Cuts KV‑Cache Memory by 83% and Boosts LLM Speed

TurboQuant Overview

TurboQuant is a compression algorithm introduced by Google to reduce the memory footprint of the KV‑Cache in transformer‑based large language models (LLMs). The KV‑Cache stores attention keys and values for each generated token, and its size grows with three dimensions: context length, precision (bit‑width), and model size.

Existing Approaches

Sliding‑window attention: keeps only the most recent N tokens.

Linear attention: compresses the entire history into a fixed‑size hidden state, sacrificing accuracy.

Naïve quantization: reduces bit‑width but typically incurs noticeable loss.

TurboQuant Technique

TurboQuant combines two proprietary methods, PolarQuant and QJL , to quantize KV‑Cache values from 16‑bit floating point to 3‑bit integers. This yields a six‑fold reduction in cache size with virtually zero loss in model accuracy.

By shrinking the KV‑Cache, the amount of data transferred from high‑bandwidth memory (HBM) to GPU SRAM is reduced by a factor of five, decreasing data‑movement overhead and increasing inference throughput.

Experimental Results

Benchmarks on the open‑source Llama‑3.1‑8B‑Instruct model running on NVIDIA H100 GPUs show:

Up to 8× speedup on long‑context tasks compared to the uncompressed baseline.

Zero measurable degradation on standard text evaluation suites.

These results demonstrate that aggressive KV‑Cache quantization can make multi‑billion‑parameter models feasible on commodity hardware.

Practical Implications

KV‑Cache can be stored at 3‑bit precision without accuracy loss, dramatically lowering memory bandwidth requirements.

Reduced cache size translates into lower hardware costs for running LLMs locally.

The technique is applicable to any transformer model that uses a KV‑Cache, provided the model’s inference pipeline integrates the PolarQuant and QJL modules.