DeepDive into DeepSeek‑V4: Efficient Million‑Token Context, Hybrid Attention, and Muon Optimizer

Overview

DeepSeek‑V4 introduces two model variants: DeepSeek‑V4‑Flash (284 B parameters) and DeepSeek‑V4‑Pro (1.6 T parameters). Both achieve multi‑million‑token context with a 10×‑27× reduction in KV‑cache size and FLOPs compared to DeepSeek‑V3.2, while retaining comparable performance on coding and agent benchmarks.

Hybrid attention architecture

The core is a hybrid attention that interleaves Compressed Sparse Attention (CSA) , Heavily Compressed Attention (HCA) and a Sliding‑Window Attention (SWA) branch.

CSA : groups m=4 tokens, learns absolute‑position encodings, applies a linear projection, softmax weighting, RMSNorm, RoPE and FP8/FP4 quantization. Only the top‑ k compressed entries are kept; SWA provides a 128‑token local context.

HCA : uses a larger compression ratio m'=128 (no overlap) and then performs dense attention on the compressed tokens.

SWA : a causal 128‑token window added to every layer.

Compression reduces the effective sequence length by n/m for CSA and n/m' for HCA. For a 1 M‑token context, DeepSeek‑V4‑Pro uses ~4.5 GB KV cache vs ~47 GB in V3.2 (≈10× saving). FLOPs per token drop from ~91 G to ~13 G for 32 K tokens, with larger gains for longer sequences.

KV‑cache layout

Compressed KV cache stored in FP8.

Positional cache (RoPE) stored in BF16.

Indexer cache (FP4) for CSA scoring.

Temporary FP32 buffers for overlap handling.

MoE and other architectural changes

Retains DeepSeek‑MoE with 256 (Flash) / 384 (Pro) routing experts and 6 activation experts.

Gating function changed from Sigmoid to Sqrt(Softplus) to avoid gradient saturation.

Hash‑based MoE routing for the first few transformer blocks.

Multi‑Token Prediction (MTP) unchanged from V3.

Manifold‑Constrained Hyper‑Connections (mHC) replace standard residual links. mHC constrains the residual matrix to the Birkhoff polytope (doubly‑stochastic matrices) via Sinkhorn‑Knopp, guaranteeing non‑expansive updates.

Muon optimizer

All 2‑D weight matrices (Linear, Conv2d, attention Q/K/V projections) are updated with Muon; embeddings, biases, RMSNorm scales and output heads use AdamW.

Muon replaces element‑wise moments with a matrix‑wise update that stays on the Stiefel manifold (semi‑orthogonal matrices), keeping the update’s spectral norm ≤ 1.

G = weight
G /= (G.norm() + eps)          # normalize
if G.shape[0] > G.shape[1]:
    G = G.T
for _ in range(10):
    A = G @ G.T
    B = b*A + c*A@A            # (b,c) = (3.4445,-4.7750) for first 8 steps, (0.2,0.2) for last 2
    G = a*G + B @ G            # a = 0.2
if G.shape[0] > G.shape[1]:
    G = G.T

Key properties:

Operates on the Stiefel manifold, preserving orthogonality of updates.

Reduces memory: only one momentum buffer vs two in AdamW.

Weight decay and RMS rescaling (to 0.2‑0.4) are applied to match AdamW’s update magnitude.

Implemented with BF16 GEMM kernels; the bulk of work consists of matrix‑matrix multiplications that map efficiently to GPU Tensor Cores.

Efficiency results

KV‑cache usage (GB) for DeepSeek‑V4‑Pro vs DeepSeek‑V3.2:

8 K tokens: 55 MB vs 368 MB (≈6.6× reduction).

32 K tokens: 159 MB vs 1 472 MB (≈9.2×).

128 K tokens: 576 MB vs 5 887 MB (≈10.2×).

1 M tokens: 4 464 MB vs 47 092 MB (≈10.5×).

FLOPs per token (Giga‑FLOPs) for selected lengths:

8 K: V3.2 91.7 G → V4‑Pro 109 G (≈0.84×).

32 K: V3.2 115 G → V4‑Pro 13.7 G (≈1.01×).

128 K: V3.2 209 G → V4‑Pro 132 G (≈1.59×).

1 M: V3.2 1 083 G → V4‑Pro 299 G (≈3.63×).

Mathematical perspective

The hybrid attention can be interpreted as a multi‑scale simplicial approximation of the full Nerve of the token category:

SWA builds a high‑resolution local 0‑simplex (edges) subcomplex.

HCA constructs a low‑resolution quotient‑category Nerve (blocks of 128 tokens) preserving global connectivity.

CSA sparsifies the intermediate Nerve while maintaining homotopy equivalence, selecting top‑k “skeletal” edges via the Lightning Indexer.

Alternating CSA and HCA yields a hierarchical, multi‑scale representation that captures both short‑range syntax and long‑range semantics with bounded computational cost.

Conclusion

DeepSeek‑V4‑Flash and V4‑Pro demonstrate that careful architectural compression (CSA/HCA), efficient KV‑cache layouts, and the Muon optimizer together enable million‑token context at a fraction of the memory and FLOPs of prior large language models, while preserving strong performance on coding, reasoning and agent tasks.