Understanding the Full LLM Inference Pipeline: From Tokenization to Streaming Output

Each call to an LLM’s generate() on a single GPU passes through two distinct compute stages: prefill , which processes the prompt and is compute‑bound, and decode , which generates tokens one by one and is memory‑bandwidth‑bound.

1. Tokenization and Embedding

Byte‑Pair Encoding (BPE) tokenizers convert raw text into integer IDs, typically using a vocabulary of about 50,000 tokens. Each ID indexes a row in an embedding table of shape [vocab_size, hidden_dim]; for a model with a 4,096‑dimensional hidden size, each token becomes a 4,096‑dimensional vector. Position information is added at this stage, often using Rotary Position Embeddings (RoPE), which encode positions by rotating the embedding vectors.

prompt = "How does inference work?"
ids = tokenizer.encode(prompt)
# ids -> [2437, 1374, 32278, 670, 30]

# embedding_table has shape [vocab_size, hidden_dim]
vectors = embedding_table[ids]  # shape: [num_tokens, 4096]

2. Transformer Layers

The embedded sequence passes through a stack of transformer layers (typically 32–80+ layers). Each layer performs two operations:

Self‑attention computes three learned projections for each token: query Q, key K, and value V. The query of each token is scored against all keys, scaled, softmaxed, and used to weight the values.

# scores: how much each token attends to every other token
Q, K, V = x @ Wq, x @ Wk, x @ Wv
scores = (Q @ K.T) / sqrt(d_k)
weights = softmax(scores)
attn_output = weights @ V

Feed‑forward network (FFN) applies a two‑layer MLP independently to each token’s vector.

After the final layer, the hidden state of the last token is projected back to the vocabulary dimension [hidden_dim, vocab_size], a softmax is applied, and the first output token is sampled.

3. Prefill (Compute‑Bound Phase)

Prefill processes the entire prompt in parallel. All tokens compute Q, K, V simultaneously, and attention runs as a large matrix‑multiply. The key metric is Time‑to‑First‑Token (TTFT). During this phase the KV cache is populated: each layer’s K and V tensors are stored in GPU memory for reuse.

# Prefill: process the whole prompt in one shot
hidden = embed(prompt_tokens) + positions
for layer in model.layers:
    Q, K, V = project(hidden)  # for ALL tokens at once
    hidden = attention(Q, K, V) + hidden
    hidden = feedforward(hidden) + hidden
    cache_kv(layer, K, V)  # save for later
first_token = sample(project_to_vocab(hidden[-1]))

4. Decode (Memory‑Bound Phase)

After the first token, generation switches to a token‑by‑token loop. For each new token, only its Q, K, V are computed; the cached K and V from previous tokens are reused. The metric here is Inter‑Token Latency (ITL), the time between consecutive output tokens.

# Decode: one token per iteration
token = first_token
steps = 0
while token != STOP and steps < MAX_STEPS:
    x = embed(token) + position(steps)
    for layer in model.layers:
        q, k, v = project(x)
        K_all, V_all = caches[layer].append(k, v)  # cached history + new
        x = layer.forward(q, K_all, V_all, x)
    token = sample(project_to_vocab(x))
    steps += 1
    yield token

5. KV Cache

Without a KV cache, generating a 1,000‑token answer would require recomputing attention over an ever‑growing sequence, leading to quadratic complexity. The cache stores each layer’s K and V once and appends new entries as generation proceeds. For long outputs, this can provide a 5× speedup or more.

The cache grows linearly with sequence length; a 13B‑parameter model consumes roughly 1 MiB per token, so a 4K‑token context occupies about 4 GiB of VRAM, limiting batch capacity.

Common mitigations include INT8/INT4 quantization of the cache, sliding‑window attention, grouped‑query attention (GQA), and PagedAttention (used by vLLM) which manages cache memory in fixed‑size blocks to eliminate fragmentation.

6. Attention Redesign Around the Cache

DeepSeek V4 Preview (released 2026‑04‑24) redesigns attention to reduce cache size. It combines two compressed‑attention mechanisms:

Compressed Sparse Attention (CSA) first applies softmax‑gated pooling to shrink KV entries by 4×, then performs sparse attention on the compressed tokens.

Heavily Compressed Attention (HCA) merges 128 KV entries into a single compressed entry and runs dense attention on these representations.

On a 1 M‑token context, V4‑Pro uses only 27 % of the FLOPs per token and 10 % of the KV cache compared to DeepSeek‑V3.2. In absolute terms, bf16 inference on 1 M tokens requires 9.62 GiB of KV cache, versus an estimated 83.9 GiB for the V3.2‑style architecture. Additional fp4/fp8 quantization can halve the cache again.

7. Quantization

Training typically uses FP32 or BF16 for gradient stability, but inference can tolerate lower precision. Memory savings scale linearly with bit‑width:

7B parameters, FP32 → 28 GB

7B parameters, FP16/BF16 → 14 GB

7B parameters, INT8 → 7 GB

7B parameters, INT4 → 3.5 GB

INT4 enables a 7B model to run on a laptop GPU with 4–6 GB VRAM. Methods such as GPTQ and AWQ use per‑channel scaling to minimize quality loss. Good INT4 implementations typically lose only 1–2 % on standard benchmarks. Moving from FP16 to INT8 usually halves inference latency with negligible quality impact, making quantization the highest‑yield optimization for most deployments.

8. Inference‑Service Infrastructure

Modern inference servers add optimizations outside the prefill‑decode loop:

Continuous batching interleaves tokens from multiple requests within a single GPU step, keeping utilization high even during the memory‑bound decode phase.

Speculative decoding runs a small draft model to propose several tokens, then validates them with the large model in a single forward pass, turning many serial decode steps into one parallel verification.

PagedAttention (vLLM) manages KV cache memory in fixed‑size blocks, eliminating fragmentation and allowing a single GPU to serve dozens of concurrent users.

Frameworks such as vLLM, TensorRT‑LLM, and Text Generation Inference (TGI) combine these techniques, achieving high concurrency because decode leaves substantial compute capacity idle, which continuous batching can fill.

9. Full Inference Path

Tokenize: BPE converts text to integer IDs.

Embed: IDs become vectors; RoPE adds positional encoding.

Prefill: All input tokens are processed in parallel (compute‑bound), KV cache is filled, and the first token is produced.

Decode loop: Each step generates one token, re‑projects Q, attends to cached K/V, runs FFN, samples, and appends new K/V to the cache (memory‑bound).

Detokenize: Token IDs are mapped back to text and streamed out.

10. Practical Conclusions

Long prompts mainly affect TTFT (prefill cost).

Long outputs mainly affect ITL (decode cost).

The two phases consume different hardware resources.

Context length inflates KV cache, reducing batch capacity.

During decode, GPU utilization can drop to ~30 % because the bottleneck is memory bandwidth, not compute.

Solutions focus on faster memory, smaller caches, or better batching rather than adding more compute.

When a model feels slow, the first diagnostic step is to determine whether the latency is in startup (prefill‑bound, optimize TTFT) or in streaming output (decode‑bound, optimize ITL).