Why Deploying DeepSeek‑V4 Locally with vLLM Is So Challenging

Official deployment recipe and hardware threshold

DeepSeek V4 provides two model variants: V4‑Flash (284 B) and V4‑Pro (1.6 T). The minimal single‑node deployment for V4‑Flash runs on four B200/B300 GPUs with the following Docker command:

docker run --gpus all \
  --ipc=host -p 8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  vllm/vllm-openai:deepseekv4-cu130 deepseek-ai/DeepSeek-V4-Flash \
  --trust-remote-code \
  --kv-cache-dtype fp8 \
  --block-size 256 \
  --enable-expert-parallel \
  --data-parallel-size 4 \
  --compilation-config '{"cudagraph_mode":"FULL_AND_PIECEWISE", "custom_ops":["all"]}' \
  --attention_config.use_fp4_indexer_cache=True \
  --tokenizer-mode deepseek_v4 \
  --tool-call-parser deepseek_v4 \
  --enable-auto-tool-choice \
  --reasoning-parser deepseek_v4

V4‑Pro uses the same command line except --data-parallel-size 8 and runs on eight B200 or B300 GPUs.

Image tag deepseekv4-cu130 (CUDA 13.0 required).

KV cache dtype is FP8; --block-size 256 is a hard requirement.

FP4 indexer cache is enabled via --attention_config.use_fp4_indexer_cache=True, a V4‑specific switch.

Tokenizer, tool‑call parser, and reasoning parser are set to the new deepseek_v4 parser; the older V3 parser is incompatible.

For H200 nodes the recipe includes a PD‑disaggregated deployment that splits four GPUs for prefill and four for decode, connected by MooncakeConnector + RDMA, with a vllm-router for request routing:

pip install vllm-router

vllm-router --policy round_robin \
  --vllm-pd-disaggregation \
  --prefill http://localhost:8000 \
  --decode http://localhost:8001 \
  --host 127.0.0.1 \
  --port 30000 \
  --intra-node-data-parallel-size 4 \
  --kv-connector mooncake

Flash defines three inference intensity levels (Non‑think / Think High / Think Max). The Think Max level requires --max-model-len >= 393216 tokens. Example client code:

from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
model = "deepseek-ai/DeepSeek-V4-Flash"
messages = [{"role":"user","content":"What is 17*19? Return only the final integer."}]
resp = client.chat.completions.create(
    model=model,
    messages=messages,
    extra_body={"chat_template_kwargs": {"thinking": True, "reasoning_effort": "max"}},
)

Official sampling defaults are temperature = 1.0 and top_p = 1.0.

Why the attention mechanism is complicated

Two long‑context inference challenges:

KV cache memory explosion : Standard MHA/MQA KV grows linearly with context length; at 1 M tokens the memory would exceed GPU capacity. MLA reduces the growth but still leaves a large footprint.

Expensive attention computation : Even with DeepSeek Sparse Attention (DSA), 1 M‑token attention remains the dominant cost.

V4 builds on MLA with four additional layers:

Key‑Value sharing : Halves memory usage; requires an inverse RoPE operation at the attention output.

Cross‑token KV compression : c4a: 1/4 compression; each compressed token aggregates eight original tokens (stride = 4). c128a: 1/128 compression; each compressed token aggregates 128 original tokens (stride = 128).

DSA sparse selection : After compression, top‑k tokens are selected (k = 512 for c4a, k = 8192 for c128a).

Locality preservation : Sliding‑window attention (window size = 128) runs on uncompressed tokens so queries see recent local context before hitting compression boundaries.

The heterogeneous KV cache reduces bf16 KV memory for a 1 M‑token sequence to 9.62 GiB, an 8.7× reduction from the 83.9 GiB estimate for a 61‑layer V3.2‑style stack. Using the fp4 indexer and fp8 attention halves the memory again (additional 2× saving).

Memory savings introduce KV‑cache management complexity:

Prefill uses bf16 KV cache while decode uses token‑wise fp8.

Three layer types (c4a, c128a, pure SWA) coexist, requiring the allocator to handle three compression ratios.

Within a batch, different sequences may be at different compression boundaries.

The model’s native fp4 MoE weights need special handling in vLLM.

How vLLM handles the complexity

vLLM optimizes along two axes: memory management and kernel efficiency.

Memory: tight KV‑cache packing

(1) Uniform logical block size of 256 native tokens across all compression layers. For c4a this yields 64 compressed entries (256/4), for c128a it yields 2 entries (256/128). This eliminates per‑layer page‑layout branching.

(2) Compressor residual state is treated as a sliding‑window KV cache: sliding_window = compress_ratio × coff (c4 = 8, c128 = 128). This reuses block semantics for prefix caching and lets PD‑disaggregation transmit residuals as SWA without extra bandwidth.

(3) Page‑size buckets are collapsed into three sizes (max, middle, min), each with a single block pool, eliminating runtime repartitioning and fragmentation.

Max bucket : c4a main KV, SWA KV, c4a compressor state, c128a compressor state.

Middle bucket : c4 indexer KV, c4 indexer compressor state.

Min bucket : c128a main KV.

Allocation occurs once; at runtime only bucket lookups are performed, with zero runtime re‑partitioning.

Kernel: feeding the GPU

After memory layout, the remaining challenge is the many small, memory‑bound kernels on the decode path. vLLM solves this with kernel fusion and multi‑stream concurrency.

Three fusion points on the c4a decode path:

Compressor + RMSNorm + RoPE + cache write : Element‑wise operations merged into a single kernel, yielding 1.4‑3× speedup.

Inverse RoPE + fp8 quant : fp8 batched matmul on o_lora projection, saving one HBM round‑trip and delivering 2‑3× speedup.

Fused Q‑norm + KV RoPE + K insert : Horizontal fusion of query and uncompressed SWA key work, achieving 10‑20× speedup.

Multi‑stream concurrency separates three logical stages—indexer computation, main KV compression, and SWA token insertion—into independent CUDA streams. For c128a layers (no indexer) the KV compression and SWA insertion run concurrently; for c4a layers the indexer pipeline runs on its own stream while the other two stages share a second stream.

Measured end‑to‑end latency drops 5‑6% at low batch sizes, and CUDA Graphs further reduce launch overhead.

The full implementation resides in vLLM PR #40760. Future work includes DeepGEMM MegaMoE kernels and paged prefill kernels, primarily targeting NVIDIA Hopper and Blackwell GPUs. Plug‑in support already enables V4 on Huawei Ascend and Cambricon MLU via vllm‑ascend and vllm‑mlu.

Summary

Deploying V4 locally is hard not because vLLM lacks features, but because the model pushes memory efficiency to the extreme: shared KV, double‑layer compression (c4a + c128a), DSA sparse selection, and SWA locality, all stacked with fp4 indexer and fp8 attention caches. Each layer adds engineering debt that vLLM must address through sophisticated KV‑cache allocation, kernel fusion, and multi‑stream scheduling.

For consumer‑grade GPUs, running V4 locally is currently infeasible. The primary beneficiaries are teams with H200/B200 clusters that need 1 M‑token context and multi‑turn agent workflows (e.g., long‑document analysis). The vLLM blog post, official recipe, and PR provide the most direct explanation of V4’s inference details.