AdaFlash: Adaptive Speculative Decoding with Online Self-Improvement for High-Concurrency LLM Inference

Speculative decoding accelerates LLM inference by having a lightweight draft model propose tokens that a larger target model verifies in parallel. Recent diffusion draft models like DFlash compress draft generation into a single parallel forward pass, reducing per-token draft overhead by ~16× compared to autoregressive drafters. However, parallel generation introduces high variance: acceptance rates swing across domains (e.g., DFlash averages 7.09 accepted tokens on MathQA vs. 3.40 on ShareGPT with Qwen3-8B, a 2.1× gap) and decay within a single draft sequence as later positions become less reliable. Fixed draft lengths and offline weights cannot adapt to shifting request distributions in production.

AdaFlash: An Online Learning Closed Loop

AdaFlash addresses this by turning speculative decoding into an online learning system with two complementary components:

1. On-Policy Distillation (OPD) for Domain Adaptation

Each inference round yields target-model distributions and verification results. Instead of discarding this signal, AdaFlash uses it to update the serving draft model immediately. Unlike offline distillation, OPD trains on the current model's own trajectories from the live deployment domain, directly correcting errors on real requests and reducing train–serve distribution shift. Two design choices tailor OPD to diffusion drafters:

Reverse KL + Top-1 hard labels: Bidirectional attention makes draft distributions high-entropy; forward KL would force coverage of low-probability regions. Reverse KL emphasizes the target's high-probability modes, while Top-1 hard labels from the target provide stable gradient direction.

Per-vocabulary divergence clipping: Reverse KL sums over the vocabulary; a few tokens where draft probability is non-negligible but target probability is near-zero can produce outlier gradients. AdaFlash clips each vocabulary term's contribution before summation.

2. Adaptive Length Head for Token-Level Variance

A lightweight head reads the draft model's intermediate features and predicts the overall acceptance rate of the current draft sequence, converting the fixed verification length k=16 into an expected acceptance length. After verification, the true acceptance rate provides a free supervision signal to update the length head online—no extra labels or target-model calls needed. High-quality drafts keep longer verification prefixes; low-quality drafts shrink the verification budget, reserving target-model compute for tokens more likely to be accepted.

Infrastructure: Async Training & Variable-Length Batching

To deploy online learning without blocking inference, AdaFlash modifies SGLang:

Decoupled training/inference: Inference servers write trajectories to a shared buffer; a background training server computes target-model supervision signals and updates draft model + length head weights. New weights are hot-swapped; the target model never changes, and token generation never stalls.

Variable-length request packing: Traditional verification batches assume uniform length. AdaFlash packs requests with different predicted verification lengths into compact batches, using an exponential moving average to estimate next-round capacity—raising GPU utilization while avoiding OOM.

Experimental Results

Evaluated on eight benchmarks (MathQA, GSM8K, OpenCodeInstruct, CodeAlpaca, ShareGPT, Blend, MATH-500, AIME25) with target models Qwen3-8B, Qwen3-Coder-30B-A3B, Qwen3.5-9B at concurrency 1, 32, 64, 128.

Qwen3-8B: Up to 5.32× Single-Request, Sustained Advantage at 128 Concurrency

MathQA: single-request speedup 5.32×; average acceptance length rises from DFlash's 7.09 to 9.83.

At concurrency 128, AdaFlash maintains 1.15× speedup while DFlash (0.76×) and OSD (0.83×) fall below standard autoregressive decoding.

Throughput climbs continuously across online adaptation rounds, staying above the OSD baseline throughout.

Acceptance-rate gap between GSM8K and CodeAlpaca narrows (domain variance mitigated by OPD); later draft positions show the largest acceptance-probability gains (token-level variance handled by length head + online updates).

MoE Model Qwen3-Coder-30B-A3B

At concurrency 128, AdaFlash achieves 1.69× average speedup vs. DFlash 1.02×, OSD 1.14×, EAGLE-3 0.83×.

Long-Context Reasoning (32K Output Tokens)

MATH-500: 3.67× (concurrency 1), 3.52× (concurrency 32).

AIME25: 3.18× (concurrency 1), 2.77× (concurrency 32).

Online adaptation and adaptive verification remain effective for long-chain reasoning.

Ascend 910C NPU (Cross-Domain Fixed Weights, No Online Updates)

Qwen3-8B: concurrency 1 → 3.69× (DFlash 3.13×); concurrency 128 → 1.50× (DFlash 1.11×).

From Static Component to Adaptive Inference System

AdaFlash transforms speculative decoding from a fixed pipeline into a closed loop that continuously adjusts: draft model adapts to domain shifts via target-model feedback; length head shortens verification when draft quality drops; scheduler repacks batches as concurrency and verification lengths change. This is critical for agent workloads where a single task spans planning, retrieval, tool use, code execution, and verification—domains shift, traffic bursts, and memory budgets fluctuate within one session.

Open Source & Publication

Paper: "AdaFlash: Adaptive Speculative Decoding via On-Policy Distilled Diffusion Drafters" (COLM 2026 Workshop on Efficient Reasoning, Spotlight).

ArXiv: https://arxiv.org/abs/2607.19223 Code: https://github.com/ZinYY/AdaFlash Hugging Face: https://huggingface.co/AdaFlash/Qwen3-8B-AdaFlash Authors: Yu-Yang Qian, Hao-Cong Wu, Chen Chen, Jiacheng Sun, Zhenhua Dong, Peng Zhao (corresponding), Zhi-Hua Zhou (Nanjing University AI School & Huawei Foundation Model Dept.).