Why Jeff Dean Champions Speculative Decoding: The Underlying Ideas

In a 2025 Stanford AI Club talk, Jeff Dean emphasized speculative decoding, a technique developed at Google that can accelerate large language model inference by 2–3× without sacrificing accuracy.

Speculative decoding treats the decoding process from a parallel perspective. Because auto‑regressive models cannot generate multiple tokens simultaneously due to token dependencies, the method instead generates several candidate token sequences in parallel and verifies them against the target model.

Idea 1: Auto‑regressive generation is inherently sequential, but sequence verification can be parallelized. When multiple next‑token candidates exist, parallel generation becomes straightforward, and continuous batching further improves efficiency for batches of varying lengths.

Idea 2: Small (draft) models can quickly produce multiple token candidates, while the large (target) model validates them in a single parallel step. The draft model is fast but less accurate; the target model is accurate but slower. This collaboration is analogous to a “draft‑verification” pipeline.

Idea 3: Speculative decoding relies on rejection sampling to ensure that the final token distribution matches that of the target model. The draft model assigns a probability to each token sequence; the target model does the same. By accepting samples with probability α = p(x) / (C·q(x)) and rejecting the rest, the resulting distribution reproduces the target distribution.

The article cites Theorem 3.8, which shows the expected wall‑time improvement factor as (1 - α^{γ+1}) / ((1 - α)(γ·c + 1)), where c is the draft‑to‑target time ratio (typically c < 0.05) and γ is the optimal number of speculative tokens. Corollary 3.9 proves that there exists a γ achieving at least (1 + α) / (1 + c) speedup.

Idea 4: The draft‑verification framework can be extended with tree‑based verification, dynamic draft lengths, and lightweight neural modules that generate multiple future tokens in parallel, further increasing concurrency.

Idea 5: Integrating speculative decoding with other model‑fusion strategies such as cascade or stacking introduces additional complexity in KV‑cache sharing, prefill‑decode interaction, and scheduling, but also opens opportunities for further speedups.

Overall, the article concludes that while Google heavily invests in speculative decoding and related flashing techniques, the optimal pattern for large‑small model fusion remains an open research question.