DeepSeek V4: Open‑Source Bombshell That Shakes Closed‑Source AI Giants

1. Two variants and common misconceptions

DeepSeek released a preview of V4 on 2026‑04‑24 without a pre‑launch hype or a press conference. The open‑source release includes two independently pretrained MoE models:

V4‑Pro : 1.6 T total parameters, 49 B sparse‑activated parameters, default 1 M‑token context.

V4‑Flash : 284 B total parameters, 13 B sparse‑activated parameters, also default 1 M‑token context.

The author debunks two “brain‑wash” myths: (1) Flash is not a stripped‑down version of Pro; both are full‑scale MoE models differing only in scale and sparsity. (2) The 1 M‑token context is not an optional switch—both models ship with it baked in, eliminating any server‑side distinction between short‑ and long‑context modes.

Long context fundamentally changes AI workflows: a 30‑round coding agent can retain hundreds of thousands of tokens, large‑scale projects (300 files, 150 k lines) can be processed without losing cross‑file references, and massive documents (200‑page contracts, 500‑page papers) can be reasoned over without chunking.

2. Core architectural revolutions

Traditional Transformers collapse under 1 M‑token context because Prefill computation grows quadratically and KV cache exhausts GPU memory. DeepSeek V4 tackles this with three low‑level innovations:

2.1 mHC multi‑stream constrained residual

Standard residual connections act like a single elevator in a building, mixing shallow and deep information and causing gradient vanishing or explosion. mHC upgrades the residual to a “multi‑elevator + intelligent scheduling + operational constraints” system:

Multiple parallel streams keep shallow and deep information separate.

Weight scheduling adds a “valve” per layer, allocating more compute to important layers.

Random‑matrix constraints prevent elevator overload or idle trips, stabilising training.

Result: the model can be deeper and larger without exploding compute or diverging gradients.

2.2 CSA + HCA hybrid sparse attention

Standard dense attention scales with the square of token count, making 1 M‑token context prohibitively expensive. V4 combines two sparse‑attention schemes:

CSA (Compressed Sparse Attention) : compresses consecutive tokens into a “summary” and uses a fast index to attend only to the most relevant summaries.

HCA (Highly Compressed Attention) : first partitions the text into macro blocks (like a table of contents), locates the relevant blocks, then applies CSA inside the selected blocks.

This two‑stage “coarse‑to‑fine” pipeline cuts KV cache size by ~90 % and reduces inference FLOPs by ~73 %, allowing 1 M‑token context to run smoother than older models with 128 k tokens.

2.3 Muon optimizer and training tricks

Training large MoE models traditionally suffers from gradient chaos, logits explosion, and slow convergence. V4 replaces the standard pipeline with a “top‑student learning method”:

Gradient orthogonalisation decouples directions of each layer’s gradient, accelerating convergence roughly two‑fold.

Pre‑RMSNorm normalises Q and K before they enter attention, preventing dominant values from drowning out others and stabilising training.

FP4 quantisation‑aware training adapts the model to low‑precision during training, saving GPU memory and speeding up deployment without noticeable accuracy loss.

3. Engineering optimisations that fill the hardware pipeline

Beyond architecture, V4 maximises GPU utilisation through several engineering techniques:

Fine‑grained compute‑communication overlap eliminates idle “bubbles” in MoE synchronisation.

TileLang operators provide high‑performance kernels without hand‑written CUDA, and they adapt to domestic hardware.

Batch‑agnostic deterministic computation guarantees bit‑exact outputs regardless of batch slicing, preventing deployment surprises.

Custom KV cache supports heterogeneous CSA/HCA storage and can persist frequently used prompts, avoiding recomputation on repeated requests.

4. Benchmark results – the open‑source ceiling

V4’s technical gains translate into concrete performance numbers that surpass many closed‑source leaders:

Programming : SWE‑Verified 80.6 % and Codeforces score 3206, beating most open‑source peers and approaching top proprietary models.

Long‑text : MRCR‑1M score 83.5, the highest among open‑source LLMs, handling million‑token documents with ease.

Agent, math, reasoning : ranks in the first tier, with Chinese language ability claimed to be the best among domestic models.

These results demonstrate that open‑source models, when equipped with architectural innovation and aggressive engineering, can match or exceed the performance of closed‑source giants such as GPT‑5.4, Claude‑4.6, and Gemini‑3.1.

5. Conclusion

DeepSeek V4 does not rely on flashy new tricks; it refines and integrates mHC, CSA/HCA, Muon, and TileLang into a cohesive “closed‑loop system”. The release proves that open‑source LLMs can achieve top‑tier capabilities without chasing closed‑source roadmaps, making long‑context handling and high efficiency a standard feature for the community.