DSpark in DeepSeek V4 Cuts LLM Inference Latency by Up to 85%

DSpark Speculative Decoding Framework

DeepSeek V4 adds a speculative decoding framework called DSpark and releases the full‑stack codebase DeepSpec . The update targets engineering deployment rather than changes to the base model.

Speculative Decoding Principle

Speculative decoding introduces a lightweight draft model that generates candidate tokens in parallel. The target model validates the candidate batch, turning serial token‑by‑token generation into parallel verification and reducing end‑to‑end latency without altering the output distribution.

DSpark Innovations

Semi‑Autoregressive Generation : keeps the high‑throughput parallel draft while adding a lightweight serial module that models dependencies inside each block, mitigating acceptance‑rate decay of pure parallel drafts.

Confidence‑Scheduled Verification : a confidence head predicts each token’s survival probability. Combined with a hardware‑aware prefix scheduler, the system dynamically selects the optimal verification length per request, allocating compute only to tokens with the highest expected payoff.

Scheduling Mechanism

The DSpark scheduler runs asynchronously to be compatible with zero‑overhead scheduling (ZOS) and continuous CUDA‑graph replay. It uses predictions from the previous two steps to decide the current truncation length, hiding scheduling latency, preventing GPU pipeline stalls, and preserving the target model’s output distribution.

Benchmark Results

On Qwen3 series target models (4B, 8B, 14B), DSpark improves the average accepted token length by 26.7%‑30.9% over the state‑of‑the‑art autoregressive model Eagle3 and by 16.3%‑18.4% over the parallel draft model DFlash. Compared with the prior single‑token production baseline (MTP‑1), DSpark increases generation speed by 60%‑85% for Flash models and 57%‑78% for Pro models while maintaining overall throughput.

DeepSpec Toolkit

DeepSpec implements a three‑stage pipeline: data preparation, training, and evaluation.

Data preparation : downloads prompt data, runs the target model to generate answers, and builds a target cache. For the default Qwen/Qwen3‑4B configuration the cache can reach ~38 TB, requiring sufficient storage.

Training : launch with bash scripts/train/train.sh which invokes train.py on each visible GPU. Configuration is selected via config_path in the config/ directory; individual fields can be overridden with --opts or by changing target_cache_dir.

Evaluation : run with bash scripts/eval/eval.sh. The script loads the trained draft checkpoint and measures acceptance on benchmarks including GSM8K, MATH500, AIME25, HumanEval, MBPP, LiveCodeBench, MT‑Bench, Alpaca, and Arena‑Hard‑v2.

DeepSpec defaults to a single‑node 8‑GPU setup; fewer GPUs require adjusting CUDA_VISIBLE_DEVICES. Built‑in draft models are DSpark, DFlash, and Eagle3; supported target model families are Qwen3 and Gemma.

Significance

Open‑sourcing the speculative decoding stack consolidates previously scattered engineering practices into a reproducible, extensible toolkit, allowing researchers and engineers to train custom draft models on mature infrastructure without rebuilding boilerplate inference acceleration components.

Technical report: https://github.com/deepseek-ai/DeepSpec/blob/main/DSpark_paper.pdf