DeepSeek V4 Arrives After 15 Months: Hybrid Attention Beats Expectations

V4 Versions Overview

DeepSeek released two MoE model variants on 2026‑04‑24:

V4‑Pro – 1.6 trillion parameters, 49 B per‑token activation, 384 MoE experts with 6 active per token, trained on 33 trillion tokens. Targeted at matching top closed‑source models on agentic ability, world knowledge, and reasoning.

V4‑Flash – 284 billion parameters, 13 B per‑token activation, trained on 32 trillion tokens. Sacrifices some world‑knowledge capacity for faster inference and lower API cost.

Both variants natively support a 1 M‑token context window and three inference modes: Non‑Think (direct answer), Think‑High (medium‑depth reasoning), and Think‑Max (full chain‑of‑thought, recommended with >384 K context).

Hybrid Attention Architecture

The core innovation is a hybrid attention design that combines three compression‑strength mechanisms:

CSA (Compressed Sparse Attention) – Dynamically compresses the token dimension, then applies DeepSeek Sparse Attention (DSA) on the compressed matrix. In a 1 M‑token context, CSA reduces inference compute to 27 % of V3.2 and KV‑cache size to 10 %.

HCA (Heavy Compression Attention) – Merges every consecutive 128 tokens into a single KV entry, providing coarse‑grained global context with some accuracy loss.

SWA (Sliding Window Attention) – Retains high‑precision local context within the current processing window.

These three operate together: SWA handles nearby details, HCA preserves overall structure, and CSA focuses on salient middle sections. An additional component, mHC (Manifold‑Constrained HyperConnection) , augments residual connections with manifold constraints to stabilize gradient flow in deep MoE stacks.

Training Engineering Advances

Two major upgrades to the training pipeline:

Muon optimizer – Replaces AdamW for most parameters. It approximates orthogonalization of the gradient‑update matrix via Newton‑Schulz iteration. Training follows an 8‑iteration aggressive phase (large coefficient) then a 2‑iteration conservative phase (small coefficient). Embedding, head, and RMSNorm layers retain AdamW.

FP4 quantization‑aware training (QAT) – MoE expert weights are trained directly in MXFP4 precision, halving memory usage relative to FP8 and eliminating simulated quantization during inference and reinforcement‑learning stages.

To mitigate gradient explosion and routing collapse in trillion‑parameter MoE training, V4 adds:

Anticipatory Routing – Decouples backbone and router updates; routing decisions use historic parameters, breaking positive feedback between routing spikes and MoE activations.

SwiGLU clipping – Clips the linear component of SwiGLU activations to the range [-10, 10], suppressing outlier activations.

Post‑training, V4 switches from mixed reinforcement learning to on‑policy distillation : domain experts for mathematics, coding, agents, and instruction compliance are trained separately (SFT + GRPO) and then distilled into a single model.

Performance Benchmarks

Official evaluation places V4‑Pro at the top of open‑source models:

Inference – LiveCodeBench 93.5 % and Codeforces score 3206, surpassing all publicly available open models.

World knowledge – Significantly ahead of other open models, slightly behind Gemini‑Pro‑3.1.

Agentic coding – Best open‑source score; internal tests rate it higher than Sonnet 4.5 and comparable to Opus 4.6 non‑think mode.

DeepSeek notes a remaining gap to Opus 4.6 think mode, pending third‑party verification. V4‑Flash matches Pro on inference speed, lags modestly on knowledge breadth, and offers lower latency and cost due to fewer parameters and activations.

Deployment and Ecosystem

V4 supports both NVIDIA GPUs and Huawei Ascend NPUs. Fine‑grained expert parallelism has been validated on both hardware families. Service throughput is currently limited by high‑end compute; DeepSeek expects price reductions after Ascend 950 super‑nodes become widely available later in the year.

Model weights are hosted on HuggingFace and ModelScope. The API is compatible with OpenAI and Anthropic protocols. For on‑premises deployment, the Flash quantized (W8A8) version runs on an 8‑card Ascend Atlas 800 system.

Key Takeaways

The hybrid attention stack (CSA + HCA + SWA) transforms million‑token context from a costly premium feature into an affordable open‑source standard, cutting inference compute to 27 % and KV cache to 10 % of the previous V3.2 baseline. Remaining limitations include text‑only capability, performance gaps on the hardest reasoning tasks compared with the latest closed models, and Pro’s current throughput constraints.

Future improvements are tied to the rollout of Ascend 950 super‑nodes, which should lower cost and increase throughput. The architectural innovations—hybrid attention, FP4‑precision training, and on‑policy distillation—provide a concrete technical roadmap for the broader community.