How to Build a High‑Performance, Scalable LLM Inference Engine: From Paged Attention to Multi‑GPU Parallelism

Background

Deepseek‑R1’s popularity has driven demand for efficient local deployment of large language models. The article analyzes how to improve inference performance, focusing on two key metrics: throughput (tokens per second) and response time (RT), and presents a series of engineering techniques.

Design of a High‑Performance, Scalable Inference Framework

Key requirements include high throughput, low latency, and easy extensibility. A common architecture separates CPU and GPU processes to avoid the Python GIL bottleneck. The system is split into four modules: Access layer, Scheduler, Model inference, and GPU‑memory manager.

Paged Attention – Solving GPU Memory Fragmentation

GPU memory fragmentation occurs when KV‑cache blocks are repeatedly allocated and freed. Inspired by OS virtual memory, Paged Attention manages KV‑cache in fixed‑size pages with a block table, dramatically reducing fragmentation and increasing GPU utilization. Benchmarks show up to 2.7× throughput improvement over previous versions.

Radix Attention – Caching Repeated Prompt Prefixes

Many requests share identical prompt prefixes, leading to redundant KV‑cache computation. Radix Attention builds a radix tree over shared prefixes, allowing KV‑cache reuse across requests. Experiments with SGLang show 30 % latency reduction and 1.5× throughput compared to VLLM 0.5.0.

Chunked Prefill – Preventing Request Stalls

Long prompts can monopolize GPU resources during the prefill stage, causing decode‑stage stalls. By splitting prompts into fixed‑size chunks (e.g., 512 tokens) and processing each chunk sequentially, prefill and decode can run in parallel without interference. Enabling this feature in vLLM cuts max RT roughly in half under high QPS.

Shortening Output Length

Reducing the number of generated tokens directly lowers latency. Strategies include setting a lower max_tokens parameter, adding concise instructions to the prompt, or fine‑tuning the model to produce shorter answers.

Multi‑GPU Parallelism

Tensor parallelism splits model weights across GPUs, allowing simultaneous attention computation. Tests show single‑GPU inference (RT ≈ 3 s, QPS ≈ 1.2) versus dual‑GPU (RT ≈ 1.7 s, QPS ≈ 2.4), roughly halving latency and doubling throughput.

Speculative Decoding (Predictive Decoding)

A small model generates candidate tokens, which are then verified by the large model. This reduces the number of heavy inference steps. For a 70 B model, speculative decoding cuts RT from 11 s to 2.8 s.

Deploying Deepseek‑R1 with SGLang

The article provides a step‑by‑step deployment guide: download the model from ModelScope, prepare a two‑node, 2 × 8 H100 cluster, and launch SGLang servers with tensor‑parallelism. Example command:

python -m sglang.launch_server --model-path deepseek-ai/DeepSeek-V3 --tp 16 --dist-init-addr 10.0.0.1:5000 --nnodes 2 --node-rank 0 --trust-remote-code

Conclusion

Combining CPU/GPU process separation, Paged Attention, Radix Attention, chunked prefill, output‑length control, multi‑GPU tensor parallelism, and speculative decoding yields a robust, high‑throughput LLM serving stack. The author plans to continue monitoring and sharing emerging inference optimizations.