DeepSeek V4: 1M‑Token Context’s Impact on Model, Inference, Cache & Agents

DeepSeek released the preview versions DeepSeek‑V4‑Pro and DeepSeek‑V4‑Flash on Hugging Face, announcing a "cost‑effective 1M context length". The report treats the 1 M token window not as a simple size increase but as a set of intertwined engineering challenges.

Key engineering changes

Attention is split into two compressed routes: CSA (Compressed Sparse Attention) , which groups m tokens into a KV entry and selects the top‑k compressed KV for each query, and HCA (Heavily Compressed Attention) , which uses a more aggressive compression factor m' while keeping dense attention.

Residual connections are replaced by mHC (Manifold‑Constrained Hyper‑Connections) , constraining the residual matrix to a doubly‑stochastic manifold to keep the spectral norm ≤ 1 and stabilize deep stacking.

The optimizer is switched to Muon (orthogonalization via hybrid Newton‑Schulz, Nesterov trick and RMS rescaling) for most modules, keeping AdamW only for embeddings, prediction heads, mHC static bias/gating and RMSNorm.

Mixed‑precision FP4/FP8 is applied to MoE expert weights and the CSA indexer QK path, achieving roughly 2× speed‑up with 99.7 % KV‑entry recall.

These tricks together reduce per‑token FLOPs to 27 % (V4‑Pro) or 10 % (V4‑Flash) of V3.2, and KV‑cache cost to 10 % or 7 % respectively.

KV‑cache as a managed storage system

Instead of a simple paged tensor, V4 treats KV‑cache as a storage system with lifecycle, compression granularity and hit‑strategy. It introduces a hybrid layout: a classical KV part for CSA/HCA entries and a state cache for sliding‑window and uncompressed tail states, with on‑disk KV cache to reuse shared prefixes across agent requests.

Inference budgeting

V4 defines three inference modes:

Non‑think : fast, intuition‑driven answers for low‑risk tasks.

Think High : conscious logical analysis for complex planning or code review.

Think Max : maximum reasoning budget for tasks requiring ≥ 384 K context, such as difficult problem solving.

Benchmarks show the Max mode pushes V4‑Pro to high scores (e.g., SimpleQA‑Verified 57.9, LiveCodeBench 93.5), while many tasks see little gain over High, indicating that the highest budget should be used selectively.

Chinese professional (white‑collar) tasks

DeepSeek built 30 Chinese occupational tasks covering finance, law, education, etc., evaluated on task completion, instruction following, content quality and formatting. V4‑Pro‑Max wins overall with 53 % vs. 37 % for Claude Opus‑4.6‑Max, especially in analysis and generation, though Claude still leads on instruction following.

Search and agentic workflow

Two search modes are provided: a traditional RAG‑style retrieval‑augmented search for Non‑think, and an "agentic search" that allows multiple tool calls within a single reasoning loop. In internal tests, agentic search beats RAG 61.7 % vs. 18.3 % with an average of 16.2 tool calls per query, and the cost increase is marginal because the search tokens are cached in the KV system.

Code agent capabilities

V4‑Pro was evaluated on a curated set of 30 real‑world coding tasks (feature development, bug fixing, refactoring) from internal engineers. Survey results show > 90 % of respondents would use V4‑Pro as their primary coding model, though remaining issues include small mistakes, vague‑prompt misinterpretation and occasional over‑thinking.

Post‑training via on‑policy distillation

The model first trains domain experts (math, code, agent, instruction) with SFT and GRPO, then merges them into a single model using On‑Policy Distillation (OPD), replacing the mixed‑RL stage of V3.2. This yields a unified model that inherits specialized expertise while keeping a single deployment target.

Product lines and deployment guidance

Two lines are offered:

Pro (49 B activation, 1.6 T parameters) – higher quality, stronger knowledge and agent performance, suited for high‑value, high‑context tasks.

Flash (13 B activation, 284 B parameters) – lower latency and cost, suitable for high‑frequency, low‑risk tasks; with FP4 + FP8 quantization the model size drops to ~158 B.

Routing recommendations suggest using Flash with Non‑think for everyday queries, upgrading to High or Max for planning, long‑document analysis, or high‑risk design reviews, and reserving Pro for tasks demanding deep knowledge or extensive context.

Takeaways

The core insight is that making a million‑token context affordable requires a holistic redesign of attention, cache management, inference budgeting and post‑training, turning long‑context from a showcase into a cost‑controlled, production‑ready capability for agents, search and professional writing.