LLM Inference Explained: Prefill, Decode, and KV Cache Mechanics

1. The Inference Process

Large language model inference generates output token by token, which is why ChatGPT uses SSE to stream tokens for a typewriter effect. Each iteration runs a Transformer decoder-only architecture (the right half of the original encoder-decoder Transformer).

2. Prefill & Decode Stages

2.1 Tokenizer

Transformer input consists of tokens produced by a model-specific tokenizer. Tokenization converts text to tokens, then to IDs. For example, 9 Chinese characters yield 5 tokens with GPT-4o's tokenizer.

2.2 Prefill Stage

Tokens are embedded into vectors, then passed through multiple Transformer layers with self-attention. For each token, query (Q), key (K), and value (V) vectors are computed to capture contextual relationships. The model builds a KV cache storing K and V vectors for every layer and token, enabling fast lookup during decoding. Note: this KV cache is not a Redis cache but an in-memory store of K and V matrices.

Key metric: Time to First Token (TTFT) measures latency from prompt submission to first token generation.

2.3 Decode Stage

After prefill, the model enters autoregressive decoding: it uses tokens T(0) to T(n-1) to generate T(n), then T(0) to T(n) for T(n+1), continuing until max length, stop token, or end-of-sequence marker. Generated tokens are decoded back to text. Decode is memory-bound because it repeatedly reads the growing KV cache. The cache avoids recomputing all previous K/V, trading increased memory for faster inference.

Key metric: Inter-Token Latency (ITL) , also called Time Per Output Token (TPOT), measures average time between consecutive generated tokens.

Kimi's Mooncake work on KV cache optimization won the FAST 2025 best paper award.

3. Inference Frameworks

3.1 MindIE

MindIE (Mind Inference Engine) is Huawei Ascend's inference acceleration suite for full-scenario AI. It supports mainstream AI frameworks upward and diverse Ascend AI processors downward, providing multi-level programming interfaces to build inference services on Ascend hardware.

3.2 LMDeploy

LMDeploy is an efficient, user-friendly LLM deployment toolbox covering quantization, inference, and serving. Core features:

Efficient inference: Persistent Batch (Continuous Batch), Blocked K/V Cache, dynamic split/fuse, tensor parallelism, optimized kernels; claims 1.8x vLLM performance.

Reliable quantization: Supports weight and K/V quantization; 4-bit models achieve 2.4x FP16 throughput, validated via OpenCompass.

Convenient serving: Request dispatching enables multi-model inference across multiple machines and GPUs.

Stateful inference: Caches attention K/V across multi-turn conversations to avoid reprocessing history, boosting long-context multi-turn efficiency.

Excellent compatibility: Supports simultaneous use of KV Cache quantization, AWQ, and Automatic Prefix Caching.

GitHub:

https://github.com/InternLM/lmdeploy

References

1.

https://arxiv.org/abs/1706.03762

2.

https://cdn.openai.com/research-covers/language-unsupervised/language_understanding_paper.pdf

3.

https://platform.openai.com/tokenizer

4.

https://www.usenix.org/conference/fast25/presentation/qin