DeepSeek V4 (Flash & Pro) Unveils Million‑Token Context and Trillion‑Parameter Inference

DeepSeek V4 Release Overview

On April 24, 2026 DeepSeek Lab launched the fourth‑generation flagship models—DeepSeek‑V4‑Pro, focused on deep reasoning and long‑text understanding, and DeepSeek‑V4‑Flash, optimized for high‑throughput, low‑latency inference. The models push the context window to one million tokens and claim a substantial reduction in inference economics.

Core Architectural Breakthroughs

The key innovation is a hybrid attention architecture that combines Compressed Sparse Attention (CSA) with Heavily Compressed Attention (HCA). CSA compresses key‑value (KV) entries at the token level and applies DeepSeek Sparse Attention to sparsify the attention matrix, while HCA compresses KV caches at a 128:1 ratio into Multi‑Query Attention streams with a 128‑token sliding window for recent dependencies. In a 1 M‑token scenario, DeepSeek‑V4‑Pro reduces per‑token FLOPs to 27 % of V3.2 and KV memory to 10 % of V3.2, enabling ten‑fold more context on the same hardware.

Manifold‑Constrained Hyper‑Connections (mHC)

When scaling to 1.6 trillion parameters, training stability becomes critical. mHC constrains inter‑layer weight updates to a Riemannian manifold, limiting signal amplification from 3000× (unconstrained) to within 2× and preventing loss spikes. The technical report notes that the 14.8 T‑token pre‑training run experienced no unrecoverable crashes, a rarity for models of this size.

Muon Optimizer

DeepSeek V4 replaces the common AdamW optimizer with the Muon optimizer (Momentum + Orthogonalization). By orthogonalizing gradients, Muon avoids redundant updates and achieves faster convergence and better generalization in large‑scale pre‑training, reducing GPU time needed for comparable performance.

Technical Specs and Benchmark Performance

DeepSeek‑V4‑Pro contains 1.6 T total parameters (49 B active) and DeepSeek‑V4‑Flash contains 284 B total parameters (13 B active). Both support a 1 M token context window and FP4/FP8 mixed precision. Benchmark results show DeepSeek‑V4‑Pro achieving 93.5 % Pass@1 on LiveCodeBench (surpassing Claude 4.6/4.7 at 88.8 % and GPT‑5.5 at 72.8 %), 80.6 % on SWE‑bench Verified, 90.1 % on GPQA Diamond, and 87.5 % on MMLU‑Pro.

Deployment Hardware Requirements

Memory estimates for a 1 M‑token context are ~3.2 TB (BF16) or ~865 GB (FP8) for the Pro model, requiring 16‑24 × H100 (80 GB) GPUs. The Flash variant needs ~284 GB (FP8) or ~160 GB (INT4), runnable on 4 × H100 or 8 × RTX 4090 GPUs. Real‑world testing shows that even with aggressive 4‑bit quantization, running V4‑Flash on a dual RTX 5090 (32 GB) is possible only with reduced context lengths.

Software Stack Dependencies

OS: Ubuntu 22.04/24.04 (Windows via WSL2)

CUDA: ≥12.4 (12.6+ recommended for FP4 kernels)

Python: 3.11 or newer

vLLM ≥0.20.0, transformers > 4.51.1 (installed from source), xformers nightly (e.g., 0.0.33.dev20251104+cu128)

Developer Experience Highlights

Users praise the model’s “thinking mode” (Chain‑of‑Thought) for complex agent workflows, but note that setting reasoning_effort="max" can cause up to two minutes of silent latency. Cost analysis shows V4‑Flash output costs $0.28 per M tokens, over 100× cheaper than GPT‑5.5, prompting many startups to migrate.

Deployment Pitfalls

When deploying V4‑Flash with the vllm‑openai:deepseekv4‑cu129 image, inputs longer than 64 k tokens cause the service to hang due to KV memory fragmentation; the temporary workaround is to disable speculative decoding or cap max‑model‑len at 32 k. Library version mismatches (e.g., transformers 4.46.x) lead to “Unrecognized model architecture” errors, requiring installation from the HuggingFace GitHub branch. Multi‑node deployments with >8 concurrent requests and >100 k token contexts can exhaust shared memory, necessitating --shm‑size=16g or higher.

Inference Economics and ROI

Compared to a traditional 1.6 T dense model, DeepSeek‑V4‑Pro reduces per‑token FLOPs by ~3.7× and 1 M‑token memory usage by ~10×. API pricing is $3.48 per M tokens versus an estimated $20+ for comparable closed‑source models. The break‑even point for self‑hosted deployment is roughly 300‑800 million tokens per month.

Future Outlook

The release positions Chinese AI research at the forefront of architectural innovation, demonstrating that hybrid attention, mHC stability, and the Muon optimizer can achieve competitive performance on both NVIDIA and domestic accelerators. Upcoming improvements such as KV offloading are expected to further stabilize million‑token inference in late 2026.

Data aggregated from official model cards, technical reports, and benchmark sources.