vLLM & SGLang Surpass DeepSeek's DeepInfer: General Engines Prevail

General Engines Catch Up to DeepInfer in Days

DeepSeek released DeepSeek-V4.1-Flash and its production inference engine DeepInfer, claiming $38/hr per GPU and 155 trillion tokens/day per cluster. Shortly after, both SGLang and vLLM announced they had matched or surpassed DeepInfer on the same benchmarks using identical hardware and test standards.

SGLang Results

Prefill: On 4×GB300 and 4×B300, SGLang achieved 1.04× DeepInfer's prefill throughput, leading at 4K, 32K, and 64K input tokens; tied at 8K and 16K.

Decode: On a 64-GPU GB300 (EP64) cluster, SGLang reduced per-step generation latency by 1.1%–7.4% for concurrency 16–256 per GPU; matched at 8 concurrency.

vLLM Results

Cluster throughput & cost: On a single GB300 NVL72 rack, using PD separation and Wide-EP expert parallelism, vLLM hit 954K tokens/s per GPU on SemiAnalysis AgentX (P90 single-user 196 tokens/s), a 7.9× improvement over single-node.

Per-GPU revenue: At DeepSeek's published pricing, vLLM yields $38.96/hr per GPU , slightly above DeepInfer's $38 benchmark.

vs. DeepInfer curves: On GB200 (131K context, nextn=3), vLLM's decode throughput closed to within 6% of DeepInfer across batch sizes 8–256. On GB300, DEP2 prefill curve tracked DeepInfer's published line throughout.

Single-user speed: A dedicated low-latency decode path pushed single-user P90 to 826 tokens/s .

How the Speedup Was Achieved

Two core strategies: absorbing DeepSeek's open-sourced low-level operators, and extreme engineering tuned to the model's architecture.

1. Full Integration of Bottom-Up Operators: Mega Series & TileLang

DeepSeek open-sourced FlashMLA, DeepGEMM, DeepEP, DeepSelect, and TileLang. SGLang and vLLM merged them into mainline immediately. The highlight is the Mega series :

MegaMoE fuses token routing, expert GEMM, and result merging into one kernel, drastically cutting GPU memory swaps.

MegaAttention became the default attention path on Blackwell B300/GB300.

2. Exploiting Model-Specific Traits: DEP2 & Wide-EP

Ultra-light KV cache: Cross-layer KV sharing + FP4 compression reduces global KV cache to 890 bytes/token , removing memory capacity as a batch-size bottleneck.

Prefill prefers small scale (DEP2 optimal): vLLM found DEP2 (2-GPU expert+data parallel) delivers 64.4K tokens/s per GPU, while DEP4 drops to 53.3K tokens/s due to 13.6% imbalance from 4-way data parallelism.

Decode must use Wide-EP: Even at 128K context, 62% of decode step time is spent loading MoE weights; attention + indexer only 13%. Scaling context from 128K to 855K adds just 7% latency (8.93ms → 9.53ms). Wide-EP with EPLB dynamic expert load balancing is the only way to saturate multi-GPU bandwidth.

3. Algorithmic Pruning: SWA Bounded Replay & Engram Large Pages

SWA Bounded Replay: Recompute only last 128 tokens for long contexts. SGLang supports decode replay and prefix-hit prefill replay. vLLM further retrofitted Layer 20 (originally full recompute) for replay, yielding 19.9% prefill throughput gain at 32K new tokens .

Engram vocabulary placement & huge pages: SGLang uses host memory sharing (preserving HBM for KV) and cross-GPU NVLink direct reads (Peer Shards, no All-Gather). vLLM replaced /dev/shm 64KB small pages (causing GPU TLB misses) with host-NUMA cuMem 2MB huge pages , extracting another 2.8% prefill improvement .

Async scheduling overlap: While GPU executes step N, CPU prepares step N+1 scheduling, eliminating host-device bubbles. This zero-overhead design existed in SGLang v0.4 (Dec 2024); DeepInfer docs explicitly credit SGLang.

DeepInfer Is Not a Complete Production System

DeepInfer's own README states it is primarily an offline inference engine and operator library , lacking the full system layer required for modern online serving. SGLang identifies three missing production-grade components:

Industrial-grade distributed KV cache: Cross-turn, cross-user cache reuse is essential for cost. SGLang's mature RadixAttention tree cache links GPU L1, 3FS/Mooncake L3 flash storage, and host-memory L2 HiCache.

Production PD separation: Physically separate compute-heavy prefill machines from memory-heavy decode machines, connected by high-throughput RDMA. SGLang supports Mooncake, NIXL, MORI backends.

Multi-instance cluster routing & load balancing: Gateway must dispatch requests based on cache prefix hits to avoid stragglers — a core strength of general frameworks.

Multi-model & cross-hardware compatibility: DeepInfer targets a single model. vLLM and SGLang run Llama, Qwen, Kimi, GLM, etc., on NVIDIA Hopper/Blackwell (GB300, B300, B200, H200), AMD MI355X, and domestic Chinese clusters.

These form the industrial-grade foundation enterprises trust in production.

Future Outlook: Three Predictions

Prediction 1: Algorithm-Operator Co-Design Is Inevitable, But Few Vendors Will Build Full Engines

Training without inference engineering consideration is commercial suicide. Top teams (DeepSeek's FlashMLA/Mega, Moonshot's MegaKernel, Zhipu's HiSparse) now co-design models with CUDA/Triton engineers from day one. However, writing custom operators ≠ building a production distributed inference engine . Custom operators + offline benchmark scripts cover ~15% of the effort; the remaining 85% is distributed KV transfer scheduling, fault tolerance, dynamic memory paging, network optimization, API compatibility, and heterogeneous hardware adaptation. Most model vendors lack such systems teams and need not reinvent this non-core wheel.

Prediction 2: New Standard Paradigm — Vendors Open-Source Operators, General Engines Handle Industrialization

DeepSeek's release of battle-tested operators (FlashMLA, DeepGEMM, DeepEP, Mega series) as open public goods signals a mature division of labor. General infrastructures like vLLM and SGLang, with hundreds of global contributors, can absorb these operators within 72 hours and instantly empower thousands of existing enterprise clusters. This vendor-provides-cutting-edge-operators + community-provides-universal-integration model is far more efficient than every vendor maintaining a niche engine.

Prediction 3: Dual-Track Ecosystem — Edge/Enthusiast Customization vs. Cloud/Enterprise Industrial Infrastructure

Edge/enthusiast track: Tools like Strata, focused on extreme memory compression and tiered loading for consumer GPUs, will thrive on hobbyist devices because massive cluster frameworks won't deeply optimize for a single gaming card.

Cloud/enterprise track: vLLM and SGLang, as the dual cores of general inference, will remain bedrock — like Linux in OS — continuously absorbing vendors' best point innovations while sustaining multi-hardware, multi-model, large-scale cluster support.

The vitality of open source lies not in one player's solo sprint, but in the next wave merging a hundred streams within days.