Can SpargeAttn Accelerate Any Model Without Training? A Deep Dive

Background

Transformer‑style attention has quadratic time complexity O(N²), which becomes a bottleneck for long sequences. The attention matrix P is naturally sparse—many entries are close to zero—yet most sparse‑attention methods rely on fixed patterns (e.g., sliding windows) or require retraining, limiting their applicability across diverse models and tasks.

Challenges

Generality : Language, video, and image‑generation models exhibit distinct and dynamic sparsity patterns, making a universal sparse‑attention design difficult.

Accuracy vs. Overhead : Predicting the sparse region of P must be both precise (to avoid accuracy loss) and extremely fast (to keep overall speedup), a trade‑off many prior works cannot satisfy.

Method

The authors propose SpargeAttn , a training‑free sparse‑attention framework that can be applied to any model. The pipeline consists of:

Selective compression of the query (Q) and key (K) matrices, followed by a fast prediction of the sparse locations in the attention matrix P.

Application of a TopCdf operation that discards the matrix‑multiplication of the predicted sparse entries (QKᵀ and PV), eliminating unnecessary computation.

A GPU‑warp‑level online softmax that exploits the gap between the global maximum and each warp’s local maximum to skip additional PV multiplications.

Optional Hilbert‑curve reordering for vision models, which groups locally similar tokens to increase sparsity.

Integration with the quantized SageAttention engine for further acceleration.

Implementation Details

The full implementation is open‑source at https://github.com/thu-ml/SpargeAttn. After a single lightweight hyper‑parameter search, the method can be permanently enabled for any model without further training.

Experimental Results

Benchmarks on an RTX‑4090 show that SpargeAttn reaches 900 TOPS at 60 % sparsity, delivering a 4.5× speedup over FlashAttention on an A100 (which peaks at 200 TOPS). The technique works across language, video, and image‑generation models, achieving 4‑7× inference acceleration while preserving end‑to‑end accuracy. Overhead of the sparse‑region prediction is negligible across sequence lengths.

Conclusion

SpargeAttn provides a universal, training‑free sparse‑attention mechanism that significantly accelerates large‑scale models without sacrificing quality. Its highly optimized implementation makes the prediction overhead virtually invisible, enabling practical drop‑in speedups for a wide range of AI workloads.