Inside vLLM: How PagedAttention & Continuous Batching Speed Up LLM Inference

KV Cache: The Memory Bottleneck

LLMs generate tokens sequentially. Each token requires its own Key (K) and Value (V) vectors for attention. Recomputing K/V for all previous tokens at every step would be prohibitive, so frameworks cache them — this is the KV cache . It saves compute but consumes massive VRAM. For LLaMA-13B, a 2048-token request needs ~1.6 GB just for KV cache (40 layers × 4096 dims × 2 bytes × 2 for K+V ≈ 800 KB/token). An A100 40 GB holds the model weights (~26 GB) leaving only ~14 GB for KV cache, limiting concurrent requests to a handful.

Memory Fragmentation from Contiguous Allocation

Early frameworks (e.g., Orca) allocate one contiguous VRAM block per request sized for the maximum sequence length (e.g., 2048 tokens). If a request only generates 200 tokens, the remaining 1848 slots sit idle — internal fragmentation . Even if total free VRAM could fit a new request, no single contiguous block is large enough, so the request waits. Utilization drops to 20–40%.

PagedAttention: Virtual Memory for KV Cache

vLLM borrows the OS virtual-memory idea: split VRAM into fixed-size blocks (e.g., 4 tokens each). A request's logical KV cache is a chain of blocks scattered across physical VRAM. A block table maps logical block indices to physical block addresses. During attention, the kernel follows the block table to gather the needed K/V blocks. Benefits:

Only the last block of a request is partially unused (max waste = block_size − 1 tokens).

Blocks are uniform, so any free block can serve any request — no external fragmentation.

Blocks are managed in a memory pool ; allocation/release is O(1).

Key parameter: --gpu-memory-utilization (default 0.92) sets the fraction of VRAM the vLLM instance may use. The pool gets whatever remains after model weights and temporary buffers. If the pool exhausts, the scheduler preempts a request (evicts its blocks) — logged as preempted by PreemptionMode.RECOMPUTE. Frequent preemption signals the pool is too small; increase the utilization or reduce --max-model-len.

Prefix Caching: Reusing Common Prompt Prefixes

Many requests share identical system prompts (e.g., “You are a professional designer…”). vLLM computes a fingerprint (hash) for each full block. On a new request, the scheduler checks the prefix cache: matching leading blocks are reused, skipping both computation and memory allocation. Enable with --enable-prefix-caching. To maximize hit rate, place fixed content early in the prompt and variable content later.

Continuous Batching: Token-Level Scheduling

Static batching groups requests into fixed batches; short requests wait for long ones to finish. Continuous batching treats each decoding step as a scheduling point: finished requests release blocks immediately; new requests join the next step. The scheduler decides which sequences participate each step. This keeps GPU utilization high.

Long prompts can still monopolize the prefill phase. --enable-chunked-prefill (on by default when supported) splits long prefill into chunks of --max-num-batched-tokens tokens (default auto; throughput-oriented setups use ≥8192). The scheduler interleaves other requests between chunks. --max-num-seqs caps concurrent sequences per step (default 256 on A100/4090, 1024 on H100/H200).

Quantization: Reducing Weight & KV Precision

Model weights default to FP16 (2 bytes/param). Quantization compresses to lower precision (FP8 = 1 byte, AWQ, GPTQ). VRAM halves; compute speeds up due to reduced bandwidth and tensor-core support. Author notes quantization gives the single largest performance gain. Parameters: --quantization fp8_per_tensor — quantize weights to FP8 on the fly. --kv-cache-dtype fp8 — store KV cache in FP8, doubling token capacity per block.

GPU Worker & EngineCore: Multi-GPU Orchestration

A GPU worker process owns one GPU: loads weights, runs PagedAttention kernels, manages its local block pool. For models exceeding single-GPU VRAM (e.g., DeepSeek-V2), vLLM offers two parallelism modes:

Tensor Parallelism ( --tensor-parallel-size N): split each layer's matrices across N GPUs. Requires fast interconnect (NVLink) for per-layer all-reduce.

Pipeline Parallelism ( --pipeline-parallel-size N): split layers into N stages; each GPU holds consecutive layers. Less communication but higher latency (sequential stages).

Multiple workers need a central coordinator: EngineCore runs the scheduler, block manager, prefix cache, and preemption logic. It dispatches work to workers, abstracting multi-GPU complexity.

End-to-End Request Flow

HTTP request hits API server → tokenized → sent to EngineCore.

Scheduler checks prefix cache, allocates blocks from pool, adds request to continuous batch.

Batch dispatched to GPU worker(s).

Worker runs PagedAttention: for each token, follows block table to gather K/V, computes attention, writes new K/V to next block (allocating from pool if needed).

Generated token appended; loop until EOS or max length.

All blocks returned to pool; response sent via API server.

Observability & Tuning via /metrics

Key Prometheus metrics: vllm:kv_cache_usage_perc — pool utilization (near 1 = pressure). vllm:num_requests_waiting — queue depth. vllm:num_preemptions — preemption rate (rising = pool too small).

Prefix cache hit rate = vllm:prefix_cache_hits / vllm:prefix_cache_queries.

Tuning actions:

High KV usage + waiting + preemptions → increase --gpu-memory-utilization, decrease --max-model-len, or enable --kv-cache-dtype fp8.

Low prefix hit rate → reorder prompts (fixed prefix first).

Common vLLM v0.31.0 Parameters

必知 --model — Model to serve (required).

必知 --max-model-len — Max context length (prompt + output). Default: model's max.

必知 --gpu-memory-utilization — Fraction of VRAM for this instance. Default: 0.92.

必知 --tensor-parallel-size — GPUs for tensor parallelism. Default: 1.

必知 --max-num-seqs — Max concurrent sequences per step. Default: 256 (A100/4090) / 1024 (H100/H200).

必知 --max-num-batched-tokens — Max tokens per scheduling step. Default: auto.

常用 --quantization — Weight quantization method. Default: from model config.

常用 --kv-cache-dtype — KV cache precision. Default: matches model.

常用 --dtype — Compute/weight precision. Default: auto.

常用 --enable-prefix-caching — Enable prefix cache. Default: on if supported.

常用 --enable-chunked-prefill — Chunk long prefill. Default: on if supported.

常用 --block-size — Tokens per KV block. Default: auto.

场景 --pipeline-parallel-size — Pipeline parallel stages. Default: 1.

场景 --data-parallel-size — Replica count for data parallelism. Default: 1.

场景 --cpu-offload-gb — CPU RAM for weight offload. Default: 0.

场景 --kv-cache-memory-bytes — Explicit KV cache VRAM budget. Default: unset.

场景 --enforce-eager — Disable CUDA graph capture (saves VRAM). Default: off.

场景 --trust-remote-code — Allow custom model code. Default: off.

服务 --served-model-name — Name exposed in API. Default: same as --model.

服务 --host / --port — Listen address/port. Default: 0.0.0.0 / 8000.

服务 --api-key — API key for auth. Default: none.

服务 --enable-auto-tool-choice + --tool-call-parser — Enable tool calling. Default: off.

服务 --reasoning-parser — Extract reasoning traces. Default: unset.

已删除 --swap-space — Old CPU offload for KV (removed). Error if used.

Additional notes: --model selects the model; --served-model-name is what clients see. Bind --host 0.0.0.0 for external access and always set --api-key. For data parallelism (model fits one GPU but need throughput), use --data-parallel-size. OOM errors containing To serve at least one request with the model's max seq len end with the maximum feasible length. --enforce-eager disables CUDA graphs, saving VRAM at latency cost. --dtype auto falls back to FP16 on older GPUs (V100, T4) lacking BF16 support.