Nov 4, 2026 · Artificial Intelligence

SpecPilot: 10.87x LLM Inference Speedup via Cost-Model Guided Speculative Decoding

NVIDIA, UCB, Princeton, and Stanford introduce SpecPilot, a cost-model guided speculative decoding system that achieves 10.87x speedup over autoregressive decoding for Qwen3.8-27B on RTX PRO 6000, using 12-bit weight format, tree verification, and two-level verification to optimize per-token verification and batch scheduling.

12-bit weightsLLM inferenceNVIDIA

0 likes · 5 min read

SpecPilot: 10.87x LLM Inference Speedup via Cost-Model Guided Speculative Decoding

AI2ML AI to Machine Learning

Dec 27, 2025 · Artificial Intelligence

Why Jeff Dean Champions Speculative Decoding: The Underlying Ideas

Jeff Dean highlighted speculative decoding as a lossless inference acceleration technique that can boost large language model throughput by 2–3×, and the article breaks down its core concepts—including parallel token verification, draft‑target model collaboration, rejection sampling theory, and practical optimizations such as continuous batching and tree‑based verification.

Draft-Target ModelKV CacheLarge Language Models

0 likes · 8 min read

Why Jeff Dean Champions Speculative Decoding: The Underlying Ideas