SpecPilot: 10.87x LLM Inference Speedup via Cost-Model Guided Speculative Decoding

Researchers from NVIDIA, UC Berkeley, Princeton, and Stanford present SpecPilot , a system that automatically tunes speculative decoding for a given GPU and batch configuration. The paper (arXiv:2611.02501) and code are available at https://github.com/StargazerX0/SpecPilot.

Key Performance Results

RTX PRO 6000 : Qwen3.8-27B with DFlash2 achieves 10.87× speedup over autoregressive decoding (single request). The original speculative system delivers 4.46× on the same setup.

H100 : Same model and draft engine yields 3.95× to 8.86× (single request). With batching, FP8 weights, and DSpark draft, speedup ranges from 1.94× to 3.93×.

Throughput on H100 rises from ~184 token/s to 416 token/s after three optimization rounds (autoregressive baseline: 49.9 token/s; original system: 183.9 token/s).

Using only the 12-bit weight format (no speculation) gives 1.19×–1.28× speedup on single requests.

Why the Original System Falls Short

In single-request scenarios, nearly all latency comes from loading the target model's weights into the GPU. Verifying multiple draft chains adds almost no extra time. When batching, each request must compute its own attention and recurrent state, so weight loading no longer dominates and the original system's speedup drops by roughly half.

SpecPilot's Optimization Loop

Cost model identifies the component furthest from the theoretical performance ceiling.

Multiple agents propose independent modifications.

A controller accepts only changes that remain faster on previously unseen requests.

Every output token is verified against the original system; regressions are rejected.

Accepted changes are recorded as replayable logs, portable across draft models, target models, and GPUs.

Four Core Innovations

12-bit weight format (~12 bits/parameter): decoding happens in registers, producing bit-identical results to full-precision weights.

Tree verification on hybrid structure : moves beyond a single greedy chain to verify a tree of draft tokens.

Two-level verification : an 8-bit view scans the entire tree first; only accepted tokens undergo full-precision computation.

Batch-aware node budget : shared across requests, draft length shortened based on recent acceptance rates, and waiting requests piggyback on the same weight load.

References

Paper: https://arxiv.org/abs/2611.02501

Code: https://github.com/StargazerX0/SpecPilot

Code example

👀点击关注
AI前沿快速Follow