DeepSpec Boosts Large-Model Inference Speed by 2–5× with Speculative Decoding

Why Inference Is Slow

ChatGPT, Claude, and DeepSeek generate responses token by token, requiring a full forward pass for each token—a process known as autoregressive decoding, which limits GPU utilization.

Speculative Decoding Core Idea

DeepSpec applies speculative decoding: a lightweight draft model quickly proposes multiple candidate tokens, and the target large model validates them in parallel, passing correct guesses and correcting wrong ones.

Guess : Draft Model generates a batch of candidate tokens.

Verify : Target Model checks the tokens in parallel.

Pass : Correct tokens are accepted; incorrect ones are fixed and generation continues.

Because the target model can verify many tokens at once and the guess accuracy is high, overall inference speed improves dramatically.

DeepSpec as a Full‑Stack Framework

DeepSpec is not a single algorithm but an end‑to‑end engineering framework covering data preparation, model training, and evaluation.

1️⃣ Data Preparation

Download training data, regenerate answers with the target model, and build a cache. The default configuration for Qwen/Qwen3‑4B requires about 38 TB of storage.

2️⃣ Training

Train a draft model to imitate the output pattern of the target model. The training script launches with a single command and supports configurable algorithms and target models.

3️⃣ Evaluation

Assess speculative decoding acceptance rates on standard benchmarks such as GSM8K, Math500, AIME‑25, and HumanEval to verify actual speed gains.

Supported Algorithms

DSpark – DeepSeek’s own algorithm with a full paper.

DFlash – Block‑diffusion‑based parallel speculative decoding, accepted at ICML 2026, achieving over 6× loss‑less acceleration and 2.5× faster than the current mainstream EAGLE‑3.

Eagle3 – Third‑generation speculative decoding built on the Eagle framework.

Hardware Requirements and Getting Started

DeepSpec assumes a single node with 8 GPUs by default, but the CUDA<em>VISIBLE</em>DEVICES environment variable can limit the number of GPUs used. The codebase is written in Python and released under the MIT license.

Why This Matters

Inference cost is becoming the primary bottleneck for large‑model deployment. A 2–5× speedup under the same hardware enables serving more users, running larger models on edge devices, and delivering a markedly better real‑time interaction experience.

DeepSpec provides an industrial‑grade open‑source stack, allowing research teams to build on it without reinventing the wheel.

Try It Now

Clone the repository and explore:

git clone https://github.com/deepseek-ai/DeepSpec

Project URL: github.com/deepseek-ai/DeepSpec<br/> Stars: 1.6k+ (released 2 days ago)<br/> Language: Python<br/> License: MIT