DeepSeek Opens DSpark: A New Speculative Decoding Framework for Large Language Models

1. The Impossible Triangle of Speculative Decoding

Large‑model inference suffers from low GPU utilization because each generated token requires a full forward pass. Speculative decoding attempts to mitigate this by using a small draft model to quickly propose a candidate token sequence and then having a large target model verify the longest matching prefix and optionally add a bonus token.

The core difficulty lies in designing the draft model. Two draft architectures are compared:

Autoregressive (AR) – e.g., Eagle3 : high draft quality due to explicit token dependencies, but latency grows linearly with block length, limiting block size and network depth.

Parallel – e.g., DFlash : generates all tokens in one forward pass, so latency is independent of block length, but independent predictions cause severe suffix decay.

Parallel models can produce long blocks (γ=16) but must verify every candidate token, which under high concurrency wastes batch capacity on tokens that are likely to be rejected.

2. DSpark Architecture Overview: Hard on Both Ends

DSpark consists of two complementary components:

Semi‑Autoregressive Generation : retains the O(1) latency advantage of a parallel backbone while adding a lightweight serial head that injects local dependencies, substantially reducing suffix decay.

Confidence‑Scheduled Verification : a confidence head predicts a per‑token survival probability; a hardware‑aware scheduler dynamically selects verification length based on real‑time load, allocating target model compute to the most promising tokens.

3. Semi‑Autoregressive Generation: Teaching Parallel Models to Look Ahead

3.1 Root Cause – Multi‑Modal Collision

Parallel drafter models (e.g., DFlash) predict all positions simultaneously. When the context admits multiple plausible continuations (e.g., "of course" vs. "no problem"), independent sampling can produce mixed, nonsensical outputs such as "of problem". Acceptance rates then drop sharply for later positions.

3.2 Solution – Two‑Stage Generation

DSpark splits draft generation into:

Parallel Stage : a DFlash‑style backbone produces base logits U₁…Uᵧ and hidden states in a single forward pass.

Sequential Stage : a lightweight module injects a prefix‑dependency bias Bₖ for each position, correcting the distribution.

Two implementations of the sequential head are provided:

Markov Head : depends only on the previous token, using a low‑rank factorization B = W₁W₂ (r=256) for minimal overhead.

RNN Head : maintains a recursive state sₖ that accumulates the full prefix history, offering longer‑range dependence at slightly higher cost.

Experiments show the Markov head captures most local‑dependency benefits with negligible latency overhead.

4. Confidence‑Scheduled Verification: Avoiding Wasteful Work

4.1 Why Dynamic Verification?

Draft acceptance rates vary widely across tasks (high for code/math, low for open‑domain chat). Under light load, verifying extra tokens costs almost nothing, but under high concurrency each verification consumes batch capacity, causing other requests to queue.

4.2 Confidence Head – Predicting Prefix Survival Probability

For each draft position the model outputs a scalar confidence cₖ ∈ (0,1), representing the conditional probability that the target model will accept the token given that all previous tokens were accepted. The supervision signal is the total‑variation distance between draft and target distributions.

Because neural networks tend to be over‑confident, DSpark applies Sequential Temperature Scaling (STS) on a hold‑out validation set, calibrating cumulative survival probabilities ∏cᵢ and reducing Expected Calibration Error from 3‑8% to about 1%.

4.3 Hardware‑Aware Prefix Scheduler – Maximizing Global Throughput

The scheduler models verification length selection as a global expected‑throughput maximization problem. For R concurrent requests, each with γ confidence estimates, the prefix survival probability aᵣ,ⱼ = ∏_{i≤j} cᵣ,ᵢ is computed. The verification batch size B = Σ(1+ℓᵣ) and expected accepted tokens τ = Σ(1+ Σ_{j≤ℓᵣ} aᵣ,ⱼ) are used to maximize Θ = τ·SPS(B), where SPS(B) is the engine’s throughput curve. Because aᵣ,ⱼ is monotonic decreasing, a greedy global sort of candidate tokens by survival probability yields the optimal solution.

5. Training Objectives: Three‑Way Loss

DSpark optimizes a weighted sum of three position‑weighted losses (weight wₖ = exp(−(k−1)/γ)):

Cross‑Entropy Loss (ℒ_ce) : teaches the drafter to predict the correct next token.

Distribution‑Matching Loss (ℒ_tv) : minimizes total‑variation distance between draft and target distributions, directly improving acceptance rate.

Confidence Loss (ℒ_conf) : binary cross‑entropy that trains the confidence head to predict soft acceptance labels.

During training the target model is frozen; the draft shares the target’s embedding and LM head (also frozen) while updating only the backbone, sequential block, and confidence head.

6. Offline Experiments: Dominating the State‑of‑the‑Art

6.1 Main Results

On Qwen3‑{4B, 8B, 14B} and Gemma4‑12B, DSpark is compared against autoregressive baseline Eagle3 and parallel baseline DFlash. Average accepted length τ improves by 26.7‑30.9% over Eagle3 and 16.3‑18.4% over DFlash.

6.2 Position‑Level Analysis

Position 1: DFlash outperforms Eagle3 because parallel models can use deeper networks without O(γ) latency penalty.

Positions 2‑7: DFlash’s acceptance decays sharply, while Eagle3’s rises thanks to autoregressive conditioning.

DSpark: inherits the strong first‑token advantage of parallel models and mitigates suffix decay with the sequential head, achieving the best of both worlds.

6.3 Depth and Length Ablation

With a fixed block size of 7, increasing DSpark depth monotonically raises acceptance length; only two DSpark layers already surpass five DFlash layers, demonstrating efficient parameter utilization from injected local autoregressive dependencies.

Increasing draft length from 4 to 16 expands DSpark’s advantage over DFlash from 15‑18% to 21‑30%, because pure parallel models see diminishing returns with longer blocks while DSpark’s suffix correction remains effective.

7. Production Deployment: DSpark in DeepSeek‑V4 Services

7.1 Deployment Context

DSpark is live in DeepSeek‑V4‑Flash (preview) and DeepSeek‑V4‑Pro (preview), replacing the previous MTP‑1 single‑token speculative decoder. MTP‑1 persisted because static multi‑token drafters (MTP‑3/5) caused severe throughput degradation under high concurrency.

7.2 Throughput vs. Interactivity

Real‑world traffic shows DSpark pushes the system’s throughput‑interactivity Pareto frontier outward. Under an 80 tok/s/user SLA, V4‑Flash’s total throughput rises by 51%; under a stricter 120 tok/s/user SLA, DSpark maintains effective concurrency while per‑user speed improves 60‑85%.

V4‑Pro sees a 52% throughput gain at 35 tok/s/user and a 406% gain at 50 tok/s/user, with per‑user speed improvements of 57‑78%.

7.3 Load‑Adaptive Mechanism

Across identical concurrency levels, DSpark’s total throughput consistently exceeds MTP‑1.

DSpark automatically adjusts its average verification budget: under light load it expands to 4‑6 tokens (vs. MTP‑1’s fixed 2), while under heavy load it contracts to around 3 tokens to protect batch capacity.

DSpark: Confidence‑Scheduled Speculative Decoding with Semi‑Autoregressive Generation
https://arxiv.org/abs/2606.19348
https://github.com/deepseek-ai/DeepSpec/tree/main
https://x.com/dzhulgakov/status/2070922900400640398