SSD Framework Doubles Inference Speed Over Top Engines, Breaking the Serial Bottleneck

In the field of large language model inference, speculative decoding (SD) has become the standard acceleration technique, but it suffers from a fundamental limitation: the drafting and verification stages must be executed serially.

Researchers from Stanford, Princeton, and Together AI introduced the SSD framework and its optimized algorithm SAGUARO, which successfully parallelize drafting and verification.

Speed Claim: The algorithm achieves inference speeds up to twice as fast as the world’s strongest inference engine.

Speculative decoding works by using a small, fast draft model to guess the next tokens that a large, slower model would generate, then the large model verifies these guesses in a separate step. This process is inherently sequential.

SSD removes this sequential dependency by having the draft model predict possible verification results in advance and run speculation in parallel with verification. If any predicted verification outcome occurs, the speculative token is immediately emitted, eliminating the overhead of the drafting phase.

Challenges of Parallel Drafting and Verification

The draft model must accurately predict verification results, including how many speculative tokens will be accepted and the sampled reward token.

There is a delicate trade‑off between the acceptance rate of speculative tokens and the accuracy of verification predictions, which must be balanced to maximize speedup.

A robust fallback strategy is required to handle prediction failures, especially under large batch sizes and high temperature settings where failures are frequent.

To address these challenges, the authors introduced SAGUARO , an optimized SSD algorithm with three key innovations:

Formulating verification‑prediction as a constrained optimization problem and using the most likely draft logits to predict the reward token, achieving up to 90% accuracy.

Identifying the tension between prediction accuracy and high‑quality speculation, and developing a sampling algorithm that balances the two.

Exploring multiple fallback strategies, finding that the optimal fallback varies with batch size; with these optimizations, SAGUARO outperforms standard SD by about 20% per batch element.

Overall, SAGUARO delivers up to 2× acceleration compared to optimized speculative decoding and up to 5× acceleration over standard autoregressive generation, significantly improving the latency‑throughput Pareto frontier across various batch sizes.

Author Tanishq Kumar highlighted the impact on ultra‑long‑context inference, noting that halving latency in a data center equipped with thousands of B200 GPUs could enable double‑depth reasoning for tasks that process billions of tokens.

Future research directions include combining SSD with techniques such as EAGLE and token‑tree speculation, expanding the number of draft devices and speculative caches to further reduce latency, and exploring cluster‑level deployment of shared speculative endpoints.

Paper link: https://arxiv.org/pdf/2603.03251

GitHub repository: https://github.com/tanishqkumar/ssd