Skip to content
Tagged articles

sliding window attention

3 articles · Page 1 of 1
Machine Heart
Machine Heart
Oct 31, 2026 · Artificial Intelligence

Hybrid LLM Attention: Linear or Sliding Window? Fudan OpenMOSS Finds Both Excel

Fudan University's OpenMOSS team systematically compares sliding window and linear attention mixed with full attention in long-context LLMs, revealing complementary trade-offs: sliding window excels at length extrapolation but suffers in long-context training, while linear attention shows the opposite pattern, and proposes improvements like larger windows, LongCE loss, and sliding-window linear attention with log correction for better extrapolation.

Fudan UniversityLinear attentionOpenMOSS
0 likes · 13 min read
Hybrid LLM Attention: Linear or Sliding Window? Fudan OpenMOSS Finds Both Excel
Machine Learning Algorithms & Natural Language Processing
Machine Learning Algorithms & Natural Language Processing
Sep 4, 2026 · Artificial Intelligence

Sliding Window Attention Beats Linear Attention: Microsoft's 0-Training Baseline Matches Costly Post-Training

Microsoft research shows sliding window attention with four attention sinks achieves 99% performance recovery on knowledge tasks without any post-training, matching linear attention methods requiring hundreds of millions of tokens while outperforming them 2-10x on long-context benchmarks.

KV CacheLinear attentionLong Context
0 likes · 9 min read
Sliding Window Attention Beats Linear Attention: Microsoft's 0-Training Baseline Matches Costly Post-Training
Baobao Algorithm Notes
Baobao Algorithm Notes
Jul 31, 2024 · Artificial Intelligence

What Makes Mistral’s 7B, Mixtral, and Large 2 Models Stand Out? A Deep Technical Dive

This article compiles key technical details of the Mistral model family—including Mistral 7B, Mixtral 8×7B, Mixtral 8×22B, Mistral Nemo, and Mistral Large 2—covering their architectural innovations such as sliding‑window attention, grouped‑query attention, mixture‑of‑experts design, scaling parameters, performance benchmarks, quantization requirements, and practical deployment commands.

Grouped-Query AttentionMistralMixtral
0 likes · 17 min read
What Makes Mistral’s 7B, Mixtral, and Large 2 Models Stand Out? A Deep Technical Dive