Elastic Speculative Decoding Breaks Large‑Model Inference Bottlenecks

Large language model (LLM) inference cost dominates production serving as model sizes increase. Speculative decoding (SD) accelerates generation by running a small draft model and a large verification model in parallel, assuming that verifying multiple draft tokens costs roughly one forward pass of the target model.

Why speculative decoding fails under high concurrency

In high‑throughput serving, batch size grows and many requests compete for the verification compute of the target model. Each additional low‑value token that must be verified consumes resources, reducing overall throughput and increasing tail latency. Experiments show that for LLaMA3.3‑70B the verification compute rises with batch size and exceeds the cost of a single autoregressive step when the batch reaches 128, causing SD to underperform. Similar behavior is observed for Qwen3‑235B, where methods such as EAGLE‑3 improve throughput at low concurrency but fall below vanilla autoregressive decoding when batch size reaches 128.

ECHO: Elastic Speculative Decoding with Sparse Gating

In a batch, treat all requests’ candidate token trees as a unified Super‑Tree and allocate depth and width under a global verification budget K_max .

ECHO reframes speculative tree construction as a budget‑constrained scheduling problem. Instead of blindly increasing draft depth, it dynamically decides which requests receive more verification budget based on confidence.

Sparse Confidence Gating

Gate decisions only at the root, a target depth, and a few adaptively chosen intermediate depths (the “sweet spots” where accepted and rejected token confidence distributions are most separable).

Identify sweet spots during a warm‑up/calibration phase.

During inference, use the maximum‑likelihood path probability c_{i,d} as confidence; if c_{i,d} > \tau_d the path is deepened, otherwise it is truncated and its budget released.

Unified Elastic Budget Scheduler

The scheduler operates under the global verification budget K_max and performs two kinds of allocation:

Depth vs. width within a request : when deepening is risky, remaining budget expands the candidate set at the current depth.

Cross‑request budget reallocation : low‑confidence requests release budget that is transferred to high‑confidence requests for further deepening.

Two‑level priority rules are applied:

Priority 1 – Global Depth Extension : high‑confidence requests receive budget first to reduce the number of verification steps.

Priority 2 – Opportunistic Width Expansion : if no request qualifies for deepening, leftover budget widens the candidate set of truncated requests.

System integration with SGLang

ECHO is integrated into the industrial‑grade inference framework SGLang. The Flatten & Pack step packs irregular candidate token trees from multiple requests into a dense, kernel‑compatible layout for a single verification forward pass, eliminating ragged‑batch overhead.

Experimental evaluation

Benchmarks were run on 8 × NVIDIA H100 80 GB GPUs across models ranging from 13 B to 235 B (Vicuna‑13B, LLaMA‑3.1‑8B, LLaMA‑3.3‑70B, Qwen3‑8B/32B/235B) and tasks (HumanEval, GSM8K, CNN/DM, Alpaca, MT‑Bench). Key results:

High‑load (batch size = 256) on Qwen3‑235B: throughput rises from 2,803 tok/s to 3,207 tok/s (+14.4%).

Low‑load (batch size = 1) wall‑time speedups of 1.63×–5.35×, with a peak 5.35× on LLaMA3.3‑70B.

Compared to DDD (1.77×) and EAGLE‑3 (1.69×), ECHO achieves 2.02× on Qwen3‑235B.

On Qwen3‑32B, ECHO improves throughput by 15.8% over DDD.

Ablation studies

Two simplified variants were evaluated:

Dense Gating : gate at every layer, incurring extra overhead and mis‑predictions on unreliable depths; yields ~5% lower throughput on LLaMA3.1‑8B (batch = 256).

Fixed Threshold : a single confidence threshold for all depths; cannot adapt to depth‑dependent probability decay; on Qwen3‑235B it is 5.3% slower than full ECHO (3,046 → 3,207 tok/s).

Additional gains in the verification‑budget‑limited regime were observed: +7.92% on LLaMA3.1‑8B, +12.96% on LLaMA3.3‑70B, +10.00% on Qwen3‑8B, and +14.95% on Qwen3‑235B.

Conclusion

ECHO demonstrates that in high‑concurrency LLM serving the core of speculative decoding shifts from “guess as many tokens as possible” to “allocate a fixed verification budget to the most valuable tokens”. By modeling the problem as a global budget‑scheduling task, applying sparse confidence gating, and integrating with a production inference engine, ECHO delivers consistent throughput gains across model scales and workloads.

Paper: https://arxiv.org/abs/2604.09603