DeepSeek‑V3.2‑Exp Unveiled: Million‑Token Context, Sparse Attention, and Cost‑Effective Inference

DeepSeek‑V3.2‑Exp has been open‑sourced and its accompanying paper released, offering a wealth of technical details for researchers and developers.

There are also rumors that DeepSeek V4 will be launched in October.

🔥 Million‑level token context window

🧠 GRPO enhanced reasoning

⚡ NSA/SPCT “black‑tech” innovations

The performance of DeepSeek‑V3.2‑Exp is roughly on par with the previous DeepSeek‑V3.1‑Terminus model.

DeepSeek‑V3.2‑Exp is the latest experimental model built on V3.1‑Terminus and introduces the DeepSeek Sparse Attention (DSA) mechanism, enabling faster and more efficient training and inference on long contexts.

Paper download: https://github.com/deepseek-ai/DeepSeek-V3.2-Exp/blob/main/DeepSeek_V3_2.pdf

Sparse attention mechanism : Implements “Lightning Indexer” combined with top‑k attention to achieve sparsity.

Training foundation : Based on the V3.1 Terminus model and continues pre‑training with 1 trillion tokens.

Expert model fusion : Uses reinforcement learning to train five specialist models (e.g., coding, mathematics) and merges them into the final checkpoint via knowledge distillation.

GRPO algorithm : Employs multiple reward functions, including length penalty, language consistency, and score‑based rewards.

Performance optimization : Supports FP8 precision and sparse‑attention kernels. Relevant code can be found in the following PRs:

https://github.com/deepseek-ai/DeepGEMM/pull/200

https://github.com/deepseek-ai/FlashMLA/pull/98

https://github.com/tile-ai/tilelang/pull/894

Inference cost analysis : Although Lightning Indexer has O(L²) complexity, L is much smaller than N, dramatically reducing cost in long‑context scenarios; decoding 128K tokens costs about $0.25 versus $2.20 for dense attention, roughly a ten‑fold reduction.

https://x.com/danielhanchen/status/1972613546119991791</code><code>https://huggingface.co/deepseek-ai/DeepSeek-V3.2-Exp