Cross-Layer Sharing

AI2ML AI to Machine Learning

Oct 28, 2026 · Artificial Intelligence

Attention Evolution: From Transformer to Dynamic Sparse & Hybrid LLM Architectures

This article traces the evolution of attention mechanisms from early visual attention and RNN-based attention through the Transformer revolution to modern sparse, linear, and state-space variants, detailing recent innovations like GQA, MLA, dynamic sparse attention (MoBA, NSA), hybrid architectures, and cross-layer memory sharing in open-source LLMs such as Llama, DeepSeek, and Qwen.