Mamba’s SSD Framework Shatters Serial Bottleneck, Outperforms vLLM and SGLang

Problem: autoregressive decoding bottleneck

Large language model (LLM) inference is limited by the strictly serial nature of autoregressive token generation. Each token must be produced, verified, and then used as context for the next token, which creates a latency ceiling.

Speculative Decoding (SD) and its limitation

Speculative Decoding introduces a lightweight draft model that predicts future tokens while the target model verifies the current token. The key efficiency metric for SD is the acceptance rate , i.e., how closely the draft distribution matches the target distribution. However, SD still requires the draft model to wait for the verifier before starting the next drafting round, preserving a serial dependency.

Speculative Speculative Decoding (SSD)

SSD removes the draft‑verification dependency by deploying the draft model on an independent hardware node. While the target model validates tokens, the draft model continuously fills a speculation cache with its most likely predictions. When verification returns, a single cache lookup determines whether the token can be emitted instantly (cache hit) or whether fallback verification is needed (cache miss). The theoretical speed‑up of SSD is proportional to the cache‑hit rate, the latency of the primary verifier, and the latency of the backup predictor.

Saguaro algorithm: solving three system‑level challenges

Prediction‑cache topology optimization – Exhaustively enumerating all possible verification outcomes for a given prediction length and vocabulary size is infeasible. Saguaro casts cache allocation as a constrained optimization problem that decides the fan‑out (number of speculative branches) at each prediction depth. The optimal solution follows a truncated geometric series, assigning a larger fan‑out to shallow depths and a rapidly decaying fan‑out to deeper depths. Empirical results (Figure 4) show that this geometric fan‑out outperforms a uniform fan‑out under high‑temperature sampling.

Residual‑distribution manipulation – When a speculative token is rejected, the target model samples a “bonus” token from the residual distribution (the probability mass not covered by the draft). At high temperature the residual is hard to predict. Saguaro introduces a scaling hyper‑parameter λ that lowers the sampling probability of the highest‑frequency tokens in the draft distribution, thereby shifting probability mass to low‑frequency tokens in the residual. This increases the likelihood that the bonus token appears in the cache (Figure 5).

Batch‑size‑aware backoff strategy – Larger batch sizes raise the probability of cache misses, which would force the entire batch to block on a synchronous fallback. Saguaro defines a dynamic backoff threshold derived from the observed cache‑hit rate and system latency. When the batch size is below the threshold, a high‑precision but slower backup model is used; when the batch size exceeds the threshold, the system switches to an ultra‑low‑latency predictor (e.g., a random token generator or an n‑gram model) to avoid global slowdown (Figure 6).

Engineering optimization: sparse attention mask

To support many parallel speculation branches, SSD adds a custom sparse attention mask that guarantees strict independence of each branch while allowing them to share common verification prefixes (Figure 7). This mask enables a single forward pass of the draft model to generate all speculative branches simultaneously.

End‑to‑end evaluation

Evaluations on Llama‑3 and Qwen‑3 models, integrated with open‑source inference engines (vLLM and SGLang), demonstrate:

Up to 5× speedup over the pure autoregressive baseline.

Up to 2× improvement over highly optimized SD implementations.

These results (Figures 8‑9, Table 1) expand the Pareto frontier of latency versus throughput without modifying the underlying model architecture.

Conclusion

SSD shows that system‑level scheduling and algorithmic co‑design can break the long‑standing serial draft‑verification barrier, delivering substantial performance gains for high‑throughput, low‑latency LLM generation.

References

Paper: Speculative Speculative Decoding , arXiv:2603.03251<br/>Code: https://github.com/tanishqkumar/ssd