Skip to content
Tagged articles

Dynamic Sparse Attention

3 articles · Page 1 of 1
AI2ML AI to Machine Learning
AI2ML AI to Machine Learning
Oct 28, 2026 · Artificial Intelligence

Attention Evolution: From Transformer to Dynamic Sparse & Hybrid LLM Architectures

This article traces the evolution of attention mechanisms from early visual attention and RNN-based attention through the Transformer revolution to modern sparse, linear, and state-space variants, detailing recent innovations like GQA, MLA, dynamic sparse attention (MoBA, NSA), hybrid architectures, and cross-layer memory sharing in open-source LLMs such as Llama, DeepSeek, and Qwen.

Attention MechanismCross-Layer SharingDynamic Sparse Attention
0 likes · 17 min read
Attention Evolution: From Transformer to Dynamic Sparse & Hybrid LLM Architectures
Alibaba Cloud Developer
Alibaba Cloud Developer
Jan 15, 2026 · Artificial Intelligence

How Hierarchical Sparse Attention Breaks KVCache Limits for Ultra‑Long Context LLMs

This article explains how a hierarchical sparse‑attention framework redesigns KVCache storage across GPU, CPU, and remote memory, eliminates bandwidth and capacity bottlenecks, and enables efficient inference for 128K‑token and larger contexts with dramatically reduced GPU memory usage and higher throughput.

Dynamic Sparse AttentionGPU memory optimizationHierarchical Storage
0 likes · 20 min read
How Hierarchical Sparse Attention Breaks KVCache Limits for Ultra‑Long Context LLMs