Memory-Efficient Algorithms for Large Language Model Inference

LLM Inference Memory Wall

GPU FLOPS increase rapidly while memory bandwidth and capacity grow much slower, causing autoregressive decoding to become memory‑bound. Using the Roofline model, the work shows that short‑context decoding is compute‑heavy, but as context length grows the workload shifts to memory accesses for model weights and the KV cache. Experiments with LLaMA‑7B demonstrate a transition from weight‑dominated memory usage at 512 tokens to KV‑cache‑dominated usage at 128 K tokens.

SqueezeLLM Dense‑and‑Sparse Quantization

SqueezeLLM compresses model weights with a dense‑and‑sparse scheme. It applies sensitivity‑aware non‑uniform quantization, allocating finer quantization buckets to important values, and splits weights into a low‑bit dense component plus a sparse high‑precision component for outliers. This reduces the number of bytes read from GPU memory per token while preserving perplexity, avoiding the spikes seen with naïve uniform quantization.

KVQuant 3‑bit KV‑Cache Quantization

KVQuant targets the dynamic KV‑cache activations that grow with context length. Unlike static weight quantization, KV cache quantization must handle distribution shifts across layers, heads, token positions, and RoPE. By quantizing to 3 bits and carefully handling outliers, the memory footprint of a 128 K context is dramatically reduced with only minimal perplexity degradation.

Squeezed Attention Hierarchical Retrieval

Squeezed Attention clusters keys of a fixed context into centroids. At inference time the query first matches against coarse centroids to locate relevant clusters, then refines the search to finer clusters, computing exact attention only for the selected keys. This hierarchical retrieval reduces KV‑cache reads and attention computation without discarding useful context.

Multipole Attention Layered Approximation

Multipole Attention applies a layered approximation: nearby or important keys receive exact attention, while distant keys are represented by centroids. Implementation includes sparse FlashDecoding, centroid lookup, centroid replacement, and fast cluster updates. Experiments on Qwen3‑8B using an A6000 GPU show that increasing the sparsity ratio reduces attention latency across batch sizes compared to a dense baseline.

Conclusions

The dissertation demonstrates that memory‑efficient LLM inference requires co‑design across algorithms, numeric representations, and system implementations. Four complementary solutions are presented: (1) SqueezeLLM for weight compression, (2) KVQuant for KV‑cache compression, (3) Squeezed Attention for selective context loading, and (4) Multipole Attention for hybrid exact‑approximate attention. Together they address the memory bandwidth and capacity bottlenecks that dominate modern autoregressive LLM serving.

Code example

来源：专知
本文
约4000字
，建议阅读
5
分钟
这篇论文的价值在于把“内存效率”拆成了多个层次。