How DeepSeek‑V3.2 Cuts Inference Cost and Boosts Agent Skills with Sparse Attention

Models

DeepSeek‑V3.2 (standard) and DeepSeek‑V3.2‑Speciale (extreme‑reasoning) are released for research use only. The standard model is optimized for balanced question‑answering, general‑agent tasks and tool‑use scenarios, while the Speciale variant is tuned for high‑difficulty mathematics, coding‑contest problems and research‑level reasoning. Neither version supports tool calling in everyday dialogue.

Core Upgrade 1: DeepSeek Sparse Attention (DSA)

DSA replaces the classic O(L^2) attention with a near‑linear O(L·k) mechanism, where k (the number of selected tokens) is much smaller than the sequence length L. It consists of two components:

Lightning indexer : quickly scores relevance between a query token and all history tokens.

Fine‑grained token selection : retains only the top‑ k tokens for the actual attention computation.

The design supports FP8 precision and integrates with Multi‑Query Attention (MLA), making training more efficient. Training proceeds in two stages:

Dense warm‑up : 1,000 steps on 2.1 B tokens, training only the indexer while keeping dense attention.

Sparse stage : 15,000 steps on 943.7 B tokens, each query selects k=2048 keys.

Empirical results on 128 K token sequences show a several‑fold reduction in inference cost compared with V3.1‑Terminus. On an H800 cluster, pre‑fill cost per million tokens drops from $0.70 to $0.20 and decoding cost from $2.40 to $0.80.

Core Upgrade 2: Scalable Reinforcement Learning

The post‑training RL stage consumes more than 10 % of the total pre‑training compute budget, a rarity among open‑source LLMs. The pipeline builds on Group Relative Policy Optimization (GRPO) with three key enhancements:

Unbiased KL estimation : fixes the original K3 estimator to remove systematic bias and prevent unbounded gradient weights.

Offline sequence‑masking : computes KL between the data‑sampling policy and the current policy, masking out trajectories whose KL exceeds a threshold to avoid off‑policy noise.

Keep Routing for MoE : records the routing path during inference and forces the same path during training, stabilizing expert updates.

Training uses expert distillation: six domain‑specific expert models (math, coding, general logic, general agent, agent programming, agent search) are first trained, then their outputs generate large‑scale data for the final model. This dramatically improves performance on agent‑centric benchmarks.

Core Upgrade 3: Agent Capability Breakthrough

Context management is redesigned so that only new user messages discard historical reasoning; tool‑related messages preserve their reasoning traces. A carefully crafted system prompt forces the model to insert tool calls naturally during reasoning.

An automatic environment synthesis pipeline generated 1,827 task‑oriented environments and 85,000 complex prompts. For code agents, millions of GitHub issue‑PR pairs were filtered to create executable software‑problem environments across Python, Java, JavaScript, etc. Search agents were trained via a multi‑agent pipeline that samples long‑tail entities from web corpora and validates generated QA pairs.

Evaluation results:

SWE‑Verified: 73.1 % success rate.

Terminal Bench 2.0: 46.4 % accuracy.

Competitive scores on MCP‑Universe and Tool‑Decathlon, narrowing the gap with closed‑source models.

In summary, DeepSeek‑V3.2 combines near‑linear sparse attention, a heavyweight RL post‑training budget, and a large‑scale agent task synthesis pipeline to achieve long‑context efficiency, deeper reasoning, and robust tool‑use. The models are released for research purposes only and do not support everyday dialogue optimization.

Technical report: https://modelscope.cn/models/deepseek-ai/DeepSeek-V3.2/resolve/master/assets/paper.pdf