SpecPilot: AI Agent Framework Boosts Speculative Decoding 10.87x for Qwen3.8-27B

Existing speculative decoding systems still suffer from significant performance overhead. In single-request scenarios, target model weight loading dominates inference time, yet the number of accepted tokens per step is limited. In batched scenarios, the extra verification computation per request weakens acceleration gains. Moreover, varying model architectures, GPU hardware, and service loads make a single universal optimization impractical.

To address this, researchers from UC Berkeley, Princeton University, Stanford University, and NVIDIA open-sourced SpecPilot , a framework that lets an AI agent autonomously discover performance bottlenecks in speculative decoding and find optimization algorithms tailored to different models, GPUs, and serving scenarios — without additional training and without sacrificing output quality.

Iterative Optimization Loop

SpecPilot uses a recursive, cycle-based optimization process. Each cycle starts from the previous cycle's final system (or a stock system plus memory entries for new deployments) and consists of five phases:

Phase 0: Bottleneck Analysis

A cost model computes the gap between the current system and the theoretical performance ceiling, decomposing the ceiling into three factors:

Acceptance factor (τ factor) : gap between the current system's accepted token count and the optimal best-first tree's accepted count under the best node budget.

Kernel factor : ratio of actual step time (from profiling) to the sum of byte lower bounds for each kernel type (target model GEMM, recurrent layers, attention, drafter, prefill, etc.). The byte lower bound for a kernel is the larger of its FLOP time (at 60% peak dense compute throughput) and its byte time (at 90% peak bandwidth), measuring actual kernel efficiency vs. theoretical peak.

Overhead factor : ratio of actual step time to profiled step time, capturing overhead not covered by profiling.

At the start of each cycle, the cost model recomputes these three factors from recorded draft lattices and profiling data; the largest factor indicates the most promising optimization direction. This allows the system to dynamically identify whether the bottleneck stems from insufficient token acceptance, low kernel execution efficiency, or extra overhead outside profiling, and to follow the bottleneck as it shifts across cycles.

Phase 1: Agent Parallel Exploration

The main agent plans multiple optimization directions based on the Phase 0 cost model results, assigning independent sub-agents to run experiments in parallel on different GPUs. Each sub-agent first evaluates potential gain using the cheapest evidence (exact replay, microbenchmarks, or single-GPU timing). Only directions with an estimated net gain of at least 3% after subtracting runtime overhead proceed to algorithm implementation. The sub-agent then implements the optimization as a code patch — hidden behind an option whose default configuration still reproduces the stock system behavior. Patches must pass correctness checks and token forcing on development requests before being handed to an independent controller for performance measurement.

Phase 2: Optimization Integration

Individual optimizations may have limited standalone benefit but can combine for larger effect. A dedicated integration sub-agent stacks verified patches onto the current best system and measures the combined gain using the same evaluation rules, deciding whether to keep the combination.

Phase 3: GPU Kernel Tuning

A kernel sub-agent profiles the integrated system and prioritizes tuning the kernels farthest from their respective byte lower bounds, further reducing weight-loading and compute inefficiencies.

Phase 4: Final Validation

Each cycle ends with a sealed final test : unseen requests are used to compare the optimized system against stock, and token forcing is re-executed on outputs up to 4096 tokens. Only versions that pass both performance improvement and correctness checks are accepted; any failure rejects the version. Notably, the paper includes a fault-injection experiment that adds a logit bias after 2048 tokens. Since development outputs typically stop at 2048 tokens, only the long-output token forcing in the sealed test can trigger these faults, validating the importance of long-output token forcing for catching hidden errors.

Performance Adjudication Rules

SpecPilot isolates the modifying agent from the measuring controller. The controller runs a paired test : two systems serve identical requests on two GPUs, swapping GPUs between runs to eliminate hardware variance. A patch is adopted only if the 95% confidence lower bound of its gain exceeds 1% plus twice the stock's own timing noise (under 0.1% for H100 single-request). If the patch's output diverges from the current best system, the threshold rises to approximately 3%.

Structured Memory

Every accepted optimization is recorded as a structured memory entry containing: code diff relative to its dependencies, variant registration, correctness constraints and configuration options, experimental results, applicability limits, and untested scopes. A memory entry is retained only if applying it together with all its dependencies to the stock system exactly reproduces the source system's code. When reused for new deployments, the system re-tunes and re-measures performance but does not count reused optimizations as new discoveries. This memory system serves both subsequent iterations and cross-model/hardware migration; a new deployment's first cycle can start from stock plus the main agent's selected verified entries. As of the paper's experiments, SpecPilot had accumulated 140 verified memory entries , each with up to 12 dependencies, covering all evaluated deployment settings, including 15 entries specific to the sm_120 architecture.

Four Core Optimization Techniques

(1) W12: Lossless 12-bit Weight Format

Targeting the single-request bottleneck where target model weight loading consumes most inference time, W12 encodes each weight in 12 bits (sign bit, 4-bit exponent offset, 7-bit mantissa), exploiting the relatively concentrated exponent distribution within each output channel of BF16 weights. Out-of-range weights are recovered exactly via an additional sparse list. Unlike typical weight compression, W12 restores original weights bit-exactly at load time, leaving model computation unchanged . A dedicated GEMM kernel decodes weights directly in registers, avoiding extra decode overhead.

(2) Best-first Tree Verification for Hybrid Architectures

Traditional speculative decoding verifies only a single draft chain, often stopping early. Best-first tree verification ranks candidate nodes by their log-probability conditioned on the parent, retaining nodes with the highest cumulative path scores. For hybrid architectures containing Gated DeltaNet (GDN) recurrent layers, the team recomputes state along each root path and designed a WY verification kernel that uses tensor cores to process multiple paths in parallel .

(3) Two-Level Verification

The first round uses the nested 8-bit weight view from W12 to perform low-precision verification, producing a candidate token chain without accepting tokens or updating the target model state. The drafter then continues generating new drafts from the end of that candidate chain, and a single full-precision verification finally checks both the candidate chain and the newly generated drafts. Because only the exact verification step can accept tokens, this mechanism increases the number of tokens accepted per exact verification while remaining lossless.

(4) Three Batch Optimization Mechanisms

To mitigate the per-node verification overhead in batched requests, the team designed:

Shared Node Budget : multiple requests share a fixed verification node budget, allocated preferentially to candidates with higher path probabilities.

Dynamic Draft Length : each request dynamically selects 16, 12, or 8 draft positions based on recent token acceptance and draft confidence.

Prompt Admission : waiting prompts are split and fed into verification computation, reusing existing weight loads, but only when the scheduler would otherwise be idle.

All three mechanisms preserve the final exact verification process, improving batch inference efficiency through smarter compute allocation.

Experimental Results

Single-request scenario : SpecPilot exceeds 2x speedup over stock across all tested configurations. Notably, Qwen3.8-27B on RTX PRO 6000 achieves an average 10.87x speedup over autoregressive decoding , a substantial improvement over the original DFlash2's 4.46x.

Batched scenario : Using FP8 Qwen3.8-27B as an example, DFlash2's original average speedup over autoregressive decoding is 2.02x, which rises to 3.90x with SpecPilot ; DSpark improves from 1.94x to 3.93x .