Skip to content
Tagged articles

long-context training

2 articles · Page 1 of 1
Machine Heart
Machine Heart
Oct 31, 2026 · Artificial Intelligence

Hybrid LLM Attention: Linear or Sliding Window? Fudan OpenMOSS Finds Both Excel

Fudan University's OpenMOSS team systematically compares sliding window and linear attention mixed with full attention in long-context LLMs, revealing complementary trade-offs: sliding window excels at length extrapolation but suffers in long-context training, while linear attention shows the opposite pattern, and proposes improvements like larger windows, LongCE loss, and sliding-window linear attention with log correction for better extrapolation.

Fudan UniversityLinear attentionOpenMOSS
0 likes · 13 min read
Hybrid LLM Attention: Linear or Sliding Window? Fudan OpenMOSS Finds Both Excel
Alibaba Cloud Big Data AI Platform
Alibaba Cloud Big Data AI Platform
Jul 16, 2025 · Artificial Intelligence

ChunkFlow: Accelerating Long‑Context Model Fine‑Tuning Up to 4.5× Faster

The paper introduces ChunkFlow, an efficient training framework for variable‑length and ultra‑long sequence datasets that powers Qwen models, achieving up to 4.53× speedup over Megatron‑LM and more than 2× overall performance gains by reorganizing data into fixed‑size chunks and employing a state‑aware scheduler.

AI performanceChunkFlowGPU efficiency
0 likes · 7 min read
ChunkFlow: Accelerating Long‑Context Model Fine‑Tuning Up to 4.5× Faster