Machine Heart

Oct 31, 2026 · Artificial Intelligence

Hybrid LLM Attention: Linear or Sliding Window? Fudan OpenMOSS Finds Both Excel

Fudan University's OpenMOSS team systematically compares sliding window and linear attention mixed with full attention in long-context LLMs, revealing complementary trade-offs: sliding window excels at length extrapolation but suffers in long-context training, while linear attention shows the opposite pattern, and proposes improvements like larger windows, LongCE loss, and sliding-window linear attention with log correction for better extrapolation.