Why DeepSeek V4 Stands Apart: A Deep Dive into Its Architecture and Performance

Background and Milestones

DeepSeek, founded in 2023 in Hangzhou, progressed from the R1 release (January 2025) that matched OpenAI o1 performance to the V3 model (December 2024) with 671 B parameters and MoE architecture. A transitional V3.2 version expanded the context window to 128 K tokens. On 24 April 2026 the company unveiled V4‑Pro and V4‑Flash, both released under the MIT license.

Positioning of V4

The technical report frames V4 as a "milestone that breaks the efficiency barrier of ultra‑long context" by turning the million‑token problem from a capability issue into an efficiency issue.

Model Variants

V4‑Pro : 1.6 T total parameters, 49 B active parameters per token, 1 M token context, 384 K token maximum output, FP4 + FP8 mixed precision.

V4‑Flash : 284 B total parameters, 13 B active parameters per token, same context and output limits, FP4 + FP8 mixed precision.

Both models are open‑source and support the OpenAI ChatCompletions and Anthropic API formats.

Core Architectural Innovations

Mixed Attention (SWA → CSA → HCA) : three attention mechanisms are interleaved across transformer layers.

Sliding Window Attention (SWA) attends only the most recent 128 tokens, providing cheap local context.

Compressed Sparse Attention (CSA) compresses KV entries 4× (m=4) then selects the top‑1024 entries per query. V4‑Pro uses Top‑1024, V4‑Flash uses Top‑512. The KV compression reduces the number of accessed entries from 1 M to ~250, achieving ~1000:1 access reduction.

Heavily Compressed Attention (HCA) applies a 128:1 compression ratio and performs dense global attention on the compressed representation, compensating for CSA’s potential long‑range blind spots.

These three layers together lower inference FLOPs to 27 % (V4‑Pro) and 10 % (V4‑Flash) of the V3.2 baseline while shrinking KV cache usage to 7 % and 10 % respectively.

Manifold‑Constrained Hyper‑Connections (mHC) : standard residual connections are extended to multiple parallel streams, which caused a 3 000× signal amplification in a 27 B experiment and crashed training. By projecting the mixing matrix onto the Birkhoff polytope using ~20 Sinkhorn‑Knopp iterations, signal amplification is limited to 1.6×, enabling stable training of the 1.6 T model.

Muon Optimizer : replaces AdamW for the majority of parameters with a Newton‑style momentum update. Peak learning rate is 2.0×10⁻⁴ with cosine decay. The authors report faster convergence and better stability at trillion‑parameter scale.

FP4 Quantization‑Aware Training (QAT) : during pre‑training the model simulates FP4 (E2M1) precision for MoE expert weights while keeping the rest in FP8. This halves expert‑weight memory without noticeable quality loss. Inference uses FP4 for experts and FP8 for the remaining weights.

Training Stability Techniques

Anticipatory Routing : decouples backbone updates (θₜ) from routing index updates (θₜ₋Δₜ), breaking the feedback loop that caused loss spikes.

SwiGLU Clamping : clamps the linear component to [‑10, 10] and the gating component to ≤10, eliminating extreme activations that trigger loss spikes.

The report openly admits that the underlying theory of these tricks is not fully understood, marking them as empirical solutions that need further research.

Two‑Stage Post‑Training Paradigm

Stage 1 – Domain Expert Pre‑training : separate experts are trained on high‑quality SFT data for coding, math, agentic tasks, instruction following, world knowledge, etc., followed by GRPO (Group Relative Policy Optimization) reinforcement learning with domain‑specific reward models.

Stage 2 – Online Policy Distillation (OPD) : a unified student model generates its own outputs, then learns from the ensemble of domain experts via a reverse KL loss. This avoids the "average‑of‑all" effect of traditional multi‑task SFT.

Empirical gains include +14 % HumanEval (coding) and +27 % world‑knowledge scores compared with a single‑task baseline.

Inference Modes and Performance

Non‑Think : fast, intuition‑driven responses.

Think High : conscious logical reasoning.

Think Max : maximal reasoning depth using a strict system prompt ("Reasoning Effort: Absolute maximum …").

On the Humanity’s Last Exam (HLE) benchmark, Think Max improves V4‑Pro scores from 7.7 % (Non‑Think) to 37.7 % (≈5×), demonstrating that inference depth, not parameter count, drives the gain.

Agentic Capabilities

Introduces <|action|>, <|title|>, and <|query|> tokens for quick instruction handling without redundant prefilling.

Fully supports Claude Code, OpenClaw/OpenCode, and CodeBuddy frameworks via OpenAI‑compatible endpoints.

Tool calls and Fill‑in‑the‑Middle (FIM) are available in Non‑Think mode, enabling multi‑step agent workflows.

Infrastructure and Deployment

V4 inference is optimized for Huawei Ascend 950 series; the company announced support on launch day, while training still largely relies on NVIDIA GPUs.

LMSYS built a custom inference stack (SGLang + Miles) featuring:

ShadowRadix cache to unify three heterogeneous KV pools.

DeepGEMM Mega MoE kernel that fuses expert parallelism, FP8×FP4 GEMM, SwiGLU, and EP aggregation into a single kernel, overlapping NVLink transfers.

Flash Compressor that reduces five KV‑compression passes to a single on‑chip operation, cutting HBM round‑trips from 5 to 2.

MTP‑based speculative decoding for higher throughput.

Hardware requirements: V4‑Flash runs on a few H100 SXM GPUs; V4‑Pro needs dozens of H100/H200 GPUs.

Benchmark Results

All numbers are taken from the official DeepSeek technical report.

World Knowledge (MMLU, C‑Eval) : V4‑Pro reaches 90.1 % (MMLU) and 93.1 % (C‑Eval), surpassing V3.2 by 2–3 percentage points.

Programming (HumanEval, LiveCodeBench) : V4‑Pro scores 76.8 % (HumanEval) and 93.5 % (LiveCodeBench), the latter beating Claude Opus 4.6 (88.8 %) and Gemini‑3.1‑Pro (91.7 %).

Math (GSM8K, MATH) : V4‑Pro attains 92.6 % (GSM8K) and 64.5 % (MATH), the best among open models.

Long‑Context (LongBench‑V2) : V4‑Pro achieves 51.5 % versus V3.2’s 40.2 %.

Knowledge Retrieval (SimpleQA‑Verified) : V4‑Pro lags at 57.9 % compared with Gemini‑3.1‑Pro’s 75.6 %.

Agentic Benchmarks (Terminal Bench 2.0, BrowseComp) : V4‑Pro scores 67.9 % and 85.9 % respectively, outperforming Claude Opus 4.6 on Terminal Bench.

The report also lists areas where V4‑Pro is close but slightly behind top closed‑source models (e.g., SimpleQA, IMO‑type reasoning).

Product Strategy and Pricing

Dual‑model line: Pro (Expert Mode) for high‑depth, cost‑sensitive, large‑scale deployments; Flash (Instant Mode) for low‑latency, high‑throughput use cases.

Three access paths: web chat (no API key), DeepSeek API (model IDs deepseek-v4-pro and deepseek-v4-flash), and open‑source weights for self‑hosting.

Pricing (USD per million tokens):

V4‑Pro: $0.145 input (cache‑hit), $1.74 input (cache‑miss), $3.48 output.

V4‑Flash: $0.028 input (cache‑hit), $0.14 input (cache‑miss), $0.28 output.

Compared with Claude Opus 4.6 ($5.00 input, $25.00 output) and GPT‑5.4 ($2.50 input, $15.00 output), V4‑Flash’s output cost is 89× cheaper.

Cache‑hit pricing makes repeated system prompts effectively free, benefiting AI coding assistants, enterprise knowledge‑base Q&A, and structured agent workflows.

Open‑Source Ecosystem

MIT license permits unrestricted commercial use, modification, and redistribution.

Four Hugging Face weight sets: V4‑Flash‑Base (284 B FP8), V4‑Flash (FP4 + FP8), V4‑Pro‑Base (1.6 T FP8), V4‑Pro (FP4 + FP8).

Provided inference code, message‑encoding scripts ( encoding_dsv4.py), and conversion guides.

NVIDIA NeMo offers official fine‑tuning recipes for V4‑Flash on Slurm clusters.

Known Limitations

Architectural complexity makes future engineering harder; the authors plan to simplify in V5.

Stability tricks (Anticipatory Routing, SwiGLU Clamping) lack solid theoretical grounding.

World‑knowledge benchmarks (SimpleQA, MMLU‑Pro) still trail Gemini‑3.1‑Pro.

Retrieval quality degrades beyond 128 K tokens (≈66 % accuracy on 1 M‑token MRCR test).

V4 is labeled a preview release; further post‑training improvements are expected.

Impact on the AI Landscape

Demonstrates that algorithmic efficiency can close the gap between open‑source and closed‑source LLMs, pressuring OpenAI and Anthropic on price.

Shows a viable path for Chinese AI independence: high‑performance models built with architecture‑level innovations rather than raw NVIDIA compute.

Sets million‑token context as a default, likely spurring new applications that rely on ultra‑long memory.

Future Outlook (V5)

Simplify the V4 architecture while preserving efficiency gains.

Develop theoretical foundations for Anticipatory Routing and related stability mechanisms.

Close knowledge‑retrieval gaps and improve >128 K token retrieval fidelity.

Expand native training support for domestic chips (Huawei Ascend, etc.).

Conclusion

DeepSeek V4 proves that a 1.6 T open‑source model can deliver programming and reasoning performance comparable to top proprietary models at a fraction of the cost, shifting the AI competition from sheer scale to architectural efficiency.