Attention Evolution: From Transformer to Dynamic Sparse & Hybrid LLM Architectures
This article traces the evolution of attention mechanisms from early visual attention and RNN-based attention through the Transformer revolution to modern sparse, linear, and state-space variants, detailing recent innovations like GQA, MLA, dynamic sparse attention (MoBA, NSA), hybrid architectures, and cross-layer memory sharing in open-source LLMs such as Llama, DeepSeek, and Qwen.
