Qwen Sparse Attention Ported to Qwen3.8-27B: 2.9x Prefill, 1.8x Decode Speedup

Background: Long-Context Inference Bottleneck

In long code repositories, complex document analysis, and extended agent tasks, models can accommodate many tokens but inference speed degrades significantly as context grows.

Solution: Qwen Sparse Attention (QSA)

QSA introduces a lightweight learned indexer that compresses historical context into fine-grained information blocks, selects relevant blocks by relevance, and computes attention only on tokens within the chosen blocks. This reduces computational overhead in long-context scenarios.

Implementation Details

Base model: Qwen3.8-27B (original architecture includes full-attention layers and Gated DeltaNet layers).

QSA indexers added to 16 full-attention layers.

Selection unit: 4 tokens per block.

Fixed selection budget: 2048 tokens.

The remaining 48 Gated DeltaNet layers stay unchanged.

The adapted model is named Qwen3.8-27B-QSA . The developer (Ryu) released the llama.cpp inference engine on GitHub and model files on Hugging Face.

Benchmark Results (Cold Start, ~261K Input Tokens)

Metric                Original Qwen3.8-27B   Qwen3.8-27B-QSA   Speedup
Prefill throughput    366.84 tok/s           1071.53 tok/s       2.92x
Decode throughput     22.91 tok/s            41.07 tok/s         1.79x

Model Quality Verification

Perplexity at 32,766 tokens: original 7.072960 vs QSA 7.071384 (virtually identical).

DocVQA score: both versions achieve 50/50.

Community Quantized Version

Developer PentaCoxian created a quantized variant of Qwen3.8-27B-QSA and reported:

Context length: ~255K tokens.

Cold-start prefill: 2375 tok/s.

Decode speed: ~115 tok/s.

Peak VRAM usage: 22.8 GiB.

Significance

QSA is a technology slated for the upcoming Qwen4 release. This community port demonstrates that the core speed improvements can be realized on existing open-weight models, and the open-source ecosystem is already iterating with quantization for even higher throughput and lower memory footprint.