Why Quantization and KV‑Cache Are Key to High‑Performance LLM Inference

In 2024 an AI startup compared two inference services for the same 7B model: Service A used vLLM with INT8 quantization, while Service B ran a simple FP16 script. Although FP16 was faster for a single request, under 64 concurrent requests Service B’s latency grew 15× while Service A’s grew only 2×, and Service A achieved four times the throughput despite using half the memory.

Prefill and Decode

LLM inference consists of two distinct stages. During Prefill the model processes the entire prompt in parallel, a compute‑bound phase where GPU FLOPs dominate. During Decode the model generates tokens one by one, repeatedly reading the growing KV‑Cache; this is memory‑bound because bandwidth, not compute, limits performance.

Because Decode is bandwidth‑limited, batching many requests together can share the same weight reads, increasing the arithmetic intensity by the batch size and dramatically raising throughput.

Service Metrics

TTFT (Time To First Token) : latency until the first token, driven by Prefill.

TPOT (Time Per Output Token) : average time per generated token, driven by Decode.

Latency : TTFT + TPOT × output length.

Throughput : total tokens processed per second, the core cost metric.

Concurrency : number of simultaneous requests, limited by KV‑cache memory.

Goodput : throughput of requests that meet SLA constraints (e.g., TTFT < 2 s, TPOT < 100 ms).

SLA (Service Level Agreement) : typically defined by high‑percentile latency such as P99 TTFT < 2 s.

KV‑Cache

The KV‑Cache stores per‑layer key and value vectors for every generated token. Its size grows linearly with sequence length (e.g., a 70B model with 32K context can consume >10 GB). Because each request’s cache occupies a large fraction of GPU memory, it becomes the primary concurrency bottleneck.

Techniques such as Multi‑Query Attention (MQA) and Grouped‑Query Attention (GQA) reduce the number of KV heads, compressing the cache by a factor of 1/Nₕ or sharing heads among groups, as adopted by Llama‑2/3.

KV‑cache quantization (INT8, INT4, FP8) can further shrink memory, but the cache is more sensitive to quantization error than weights; FP8 offers a larger dynamic range suitable for cache quantization.

Continuous Batching

Traditional static batching forces all requests in a batch to start and finish together, causing severe GPU under‑utilization when request lengths differ. Continuous Batching allows new requests to join and completed ones to leave at every Decode step, keeping GPU utilization near 100 %.

Implementations (vLLM, TGI, TensorRT‑LLM) differ mainly in their scheduling policies. In workloads with high length variance, Continuous Batching can achieve 3–8× the throughput of static batching.

PagedAttention

Before PagedAttention each request pre‑allocated a contiguous KV‑cache block, leading to internal fragmentation (unused space) and external fragmentation (inability to place new blocks). PagedAttention (Kwon et al., 2023) divides the cache into fixed‑size blocks (e.g., 16 tokens) and tracks them with a block table, analogous to an OS page table.

This reduces internal fragmentation to at most one block and eliminates external fragmentation, raising cache utilization from <10 % to >90 % and increasing effective concurrency by 2–6×.

It also enables Prefix Sharing , where identical system prompts or few‑shot examples are cached once and reused across requests.

Quantization

Quantization is often misunderstood. Two dimensions matter:

Weight‑only quantization compresses only the model weights; it saves memory but does not speed up the compute‑bound Prefill stage.

Weight‑Activation quantization compresses both weights and activations, requiring INT8/INT4 matrix‑multiply kernels; this can accelerate the memory‑bound Decode stage but carries higher accuracy risk.

Two main pipelines exist:

PTQ (Post‑Training Quantization) : calibrates with a small dataset; low cost but limited precision control, especially for INT4.

QAT (Quantization‑Aware Training) : simulates quantization during training; higher precision but expensive.

Common formats are INT8 (widely supported, <1 % accuracy loss), INT4 (aggressive compression, requires careful methods such as GPTQ or AWQ), and FP8 (new standard on Hopper GPUs, larger dynamic range).

Speculative Decoding

Speculative Decoding (Leviathan et al., 2023) introduces a small draft model that generates k candidate tokens. The large model validates all k in a single forward pass; if m tokens are accepted, they are emitted without additional large‑model computation. The acceleration factor is roughly 1/(1‑p) where p is the acceptance rate.

When the draft model matches the large model 70–80 % of the time on generic text, theoretical speed‑up can reach 3.3×, but real‑world gains are 60–80 % of that. Acceptance rates drop on domain‑specific tasks, making speculative decoding potentially harmful if p falls below ~40 %.

Multi‑Token Prediction

Instead of an external draft model, approaches like Medusa (Cai et al., 2024) add multiple “heads” after the final layer, each predicting a future token. MTP (Multi‑Token Prediction) used by DeepSeek‑V3 embeds this capability inside the model, allowing the model itself to serve as the draft.

The trade‑off is increased training complexity and the need for specialized training data.

Cache and Routing

Beyond KV‑cache, systems employ:

Prefix Cache : reuse identical prompts across requests (e.g., OpenAI’s Prompt Caching).

Semantic Cache : retrieve semantically similar past completions; useful for RAG but risky for open‑ended generation.

Small‑Large Model Router : route easy requests to a 7B model and only forward hard requests to a 70B model, cutting average cost by 50–70 %.

Early Exit : stop generation after shallow layers when confidence is high; works for classification but can degrade generation quality.

Distributed Inference

When a model exceeds a single GPU’s memory, inference must be distributed. Strategies include:

Tensor Parallel (TP) : split each layer’s weight matrix across GPUs; low latency but communication grows with layer count.

Pipeline Parallel (PP) : assign different layers to different GPUs; higher latency, used for very large models.

Expert Parallel (EP) : for MoE models, route tokens to active experts; load‑balancing is critical.

Disaggregated Prefill‑Decode : separate compute pools for Prefill (compute‑bound) and Decode (memory‑bound), communicating KV‑cache between pools via high‑speed interconnects.

Edge and Private Deployment

Edge devices have strict memory limits (e.g., an 8‑GB phone can only run a 7B model after INT4 quantization, yielding ~20 tokens/s). On‑device NPUs (Apple Neural Engine, Qualcomm Hexagon) provide high INT8 throughput but require different programming models.

Privacy‑sensitive domains (medical, finance, legal) often require private deployment; however, a 70B model typically needs 4 × A100 GPUs, so quantizing to INT4 and using 2 × A100 or a distilled model becomes necessary.

Engineering Practices

Token price ≠ task cost : API pricing per token can be misleading; task‑level cost depends on request length, concurrency, and hardware amortization.

Throughput vs. latency trade‑off : increasing batch size improves throughput but raises P99 latency; dynamic batch sizing based on queue length can mitigate this.

Quantization regression : beyond benchmark MMLU loss, evaluate on real tasks (JSON generation, code, low‑resource languages) and on long‑tail samples.

Long‑context KV cost : 128K context may need 8–16 GB cache; offloading inactive cache to CPU/SSD saves memory but adds PCIe latency.

Model routing and SLA : multi‑model routing introduces additional latency components; router thresholds directly affect P99 latency.

FinOps : separate GPU cost into weight memory, KV‑cache memory, and compute; identify the crossover point where self‑hosted inference becomes cheaper than API usage.

Elastic scaling : use KEDA or similar to scale pods; cold‑start latency for loading a 70B model can be 30–60 s.

Spot instances : reduce GPU cost by 60–80 % but require stateless design to survive pre‑emptions.

Failure modes : quantization‑induced accuracy loss on specific tasks, KV‑cache OOM leading to cascade evictions, speculative decoding slowdown when acceptance rate is low, and starvation of short requests under Continuous Batching.

Conclusion

LLM inference is not a single matrix multiplication; it is a multi‑stage system where Prefill is compute‑bound and Decode is memory‑bound. Optimizations such as KV‑cache management, Continuous Batching, PagedAttention, quantization, and speculative decoding interact synergistically. A well‑designed inference stack can achieve an order‑of‑magnitude higher throughput than a naïve script, but engineers must balance latency, cost, and quality across four conflicting constraints.

Future articles will explore systematic evaluation of deployed models, including capability, safety, and alignment benchmarks across cloud, edge, and private environments.