Sliding Window Attention Beats Linear Attention: Microsoft's 0-Training Baseline Matches Costly Post-Training

Problem: Costly Linearization of Pretrained Transformers

Full attention's KV cache grows with context length. Linear attention compresses history into fixed-size recurrent states, but training from scratch is expensive. Recent works (LoLCATs, Liger-GLA, QRWKV6, SUPRA) instead convert pretrained Transformers via distillation, LoRA, or post-training — consuming 20M to 100B additional tokens.

Missing Baseline: SWA with Attention Sinks

The paper notes prior comparisons used vanilla sliding window attention (SWA) without attention sinks. When the window slides past the initial tokens, performance collapses, underestimating SWA's true capability. This work evaluates SWA(w,4): total window w, with 4 fixed positions reserved for the first attention sinks, the remaining w−4 slots retaining local history.

Knowledge & Reasoning Tasks (SWA(64,4))

Across six tasks, SWA(64,4) averages 99.0% recovery vs. the original full-attention model — nearly identical to QRWKV6's 99.1% (350M–700M post-training tokens). On MMLU, SWA reaches 93.2% vs. QRWKV6's 92.4%. Main experiments cover Phi, Mistral, Llama, Qwen, QwQ (1.3B to tens of B). In 11 model comparisons, SWA ranks highest (excluding full attention) in 9 cases; the other two trail by only 0.1 and 0.7 points.

Additional linearization tests on Qwen3-8B and two Phi-4 reasoning models: SWA averages 71.6, while Gated DeltaNet, GLA, and QRWKV6 (≈100M tokens, two-stage distillation) stall in the 50s. Phi-4 shows the same trend.

Long-Context Gap Widens

Linear attention updates a fixed-size state continuously; the challenge is deciding what to retain vs. overwrite. SWA sees only a local window per layer, but after l layers the effective receptive field expands to ~l·w, letting local information propagate depth-wise.

On Llama 3.1 8B S-NIAH (context up to 4K, windows 128/256/512), SWA meets or exceeds LoLCATs and Liger-GLA at every setting. At window 128, context 4K, S-NIAH-3 accuracy: SWA 17.2 vs. LoLCATs 1.6 ( 10.8× gap). On BABILong, LoLCATs leads slightly at short context; at 2K SWA overtakes, and at 4K the gap is 15 vs. 3 (full attention retains 60). The divergence grows with context length.

Engineering Advantages of Small Windows

Context scaled from 128 to 256K. Full attention's KV cache grows continuously; decode throughput drops sharply past ~1K. SWA's cache caps at the window size, keeping throughput stable. Tested on 4-layer Transformer, batch 1, RTX PRO 6000 Blackwell Max-Q: FlashAttention for full attention and SWA; ThunderKittens for linear attention and LoLCATs (fused kernel). SWA-64 achieves highest throughput and lowest state memory. Up to window 512, SWA's state memory matches or beats linear attention while being faster.

Conclusion

SWA + attention sinks is not new, but prior linearization work rarely compared against it directly. This paper places them side by side: zero post-training tokens vs. tens/hundreds of millions. The authors recommend trying SWA with sinks first when the goal is running an existing model under a fixed, small inference memory budget, before investing in costly post-training linearization. SWA + sinks has become a baseline future linearization methods can hardly ignore.