Attention Evolution: From Transformer to Dynamic Sparse & Hybrid LLM Architectures

Historical Review of Attention Mechanisms

The development of attention mechanisms can be divided into several key phases:

2011–2013: Visual Attention Emergence – Soft/Hard Attention, Glimpse, location weighting. Hugo Larochelle et al. used RBM/DBM for "learning where to look" with Boltzmann-distributed weights.

2014: Attention Embraces RNNs – Recurrent Attention, Softmax weights, LSTM. DeepMind's Recurrent Models of Visual Attention ; Alex Graves' Generating Sequences with RNNs (visualizing input-output weights).

2014–2015: Seq2Seq + Attention Explosion – Encoder-Decoder Attention, Alignment, Global/Local Attention, Soft vs Hard. Bahdanau et al. Neural Machine Translation by Jointly Learning to Align and Translate ; Show, Attend and Tell; Luong et al. Effective Approaches to Attention-based Neural Machine Translation .

2017: Transformer Revolution – Self-Attention, Multi-Head, Scaled Dot-Product, Positional Encoding. Vaswani et al. Attention Is All You Need – completely abandoned RNNs, using self-attention for parallel training and global dependencies.

Progress Overview (2018–2026)

After the Transformer, the focus shifted to long-context and compute efficiency, with Chinese open-source LLMs leading attention evolution:

2018–2020: Efficiency & Sparsity Exploration – Sparse Attention, Local/Global, Sliding Window. Key works: Sparse Transformer, Longformer, BigBird, SWA.

2020: Linear Attention Emerges – Kernel Feature Map, Linear Complexity, Fixed-size State. Katharopoulos et al. Linear Attention ; Performer, RetNet.

2021–2023: Rise of State Space Models (SSM) – SSM, Selective State, Parallel Training + Constant-size Decoding. S4, Mamba, Mamba-2, GLA.

2023–2024: Open-Source LLM Long-Context Explosion – GQA/MQA, MLA, KV Cache Compression, RoPE Extrapolation. Llama-3, DeepSeek-V2/V3, Qwen2 widely adopt GQA/MLA + YaRN/NTK.

2024–2025: Dynamic Sparsity + Sequence Compression – Content-aware Sparsity, Block/Token top-k, Compressed KV Entries. MoBA, NSA, DSA, MSA, CSA/HCA (DeepSeek-V4).

2025–2026: Hybrid Architectures + Controllable Memory Updates Mainstream – Layer/Head/Branch Hybrid, Gated Delta Rule, Cross-layer Reuse. Qwen3.5/Qwen3-Next (Gated DeltaNet + Gated Attention 3:1), Kimi Linear/KDA, MiniMax-M3, GLM-5; Alibaba survey proposes "Five-dimensional Memory View" and "Multi-dimensional State Memory Routing" hypothesis.

Key Technical Concepts

1. Reducing K/V Heads or Dimension Length

Reducing heads or dimension is possible but must balance expressiveness loss.

2. Random, Local, Global, Sink, Slide Relationships

Important tokens (Sink, Global) must not be missed.

Local / Slide (Sliding Window) : Each token attends only to a fixed nearby window (blue diagonal band). Core of Longformer / Mistral / Gemma local layers.

Random : Additional random connections to distant tokens (orange scatter). Provides shortcuts, reduces graph diameter, helps information propagate quickly.

Global : Few special tokens (e.g., [CLS] or first positions) attend to full sequence and are attended by all (green border). Provides global information hub.

Attention Sink : Model dumps large attention on first few tokens. Not an independent pattern but a Softmax side effect. When using pure Sliding Window, dropping early tokens causes collapse. StreamingLLM retains first few sink tokens + sliding window.

3. Slide, Delta, Gate Relationships

Dropping, modifying, forgetting are options, but pure linearity is insufficient.

Slide : Only look at recent tokens, hard cut old ones.

Delta : Use new information to "correct" old memory (precise overwrite).

Gate : Use gating to decide how much old memory to retain (soft forgetting).

Therefore, Xiaomi MiMo-V2, DeepSeek V3.2, GLM 5, Step 3.5 all adopt GQA + Linear Attention pattern.

4. Indexer/Selection, Token-Level, Block-Level Relationships

Prefer continuity; 2-gram, 3-gram generally better than token-level.

Token-level : Indexer scores each historical token, selects top-k. Most precise, picks truly relevant K/V, but scoring cost scales with sequence length. Selected blocks scattered.

Block-level : Group adjacent tokens into blocks, score and select by block. Entire selected block K/V fed to Sparse Attention. Indexer scoring reduced to 1/block-size, top-k overhead similarly reduced. Selected blocks are contiguous.

NSA → DSA → MSA → QSA belong to this series.

NSA : Balances Compress, Selection, and Sliding.

DSA : Efficient engineering of Selection.

MSA : Fuses Token + Block Level.

QSA : Fuses Gated and Selection (GDN + QSA).

5. Compress vs Heavily Compress vs Cross-layer Index Sharing vs GPU Memory Decoupling

Compression reduces token count, not heads or dimensions; pure dense compression loses too much context.

CSA : Medium compression (every m tokens → 1). Lightning Indexer + Top-k. Sparse computation. Combines compression + sparse selection. Solves long-context compute and memory.

HCA : Heavy compression (every m′ ≫ m tokens → 1). No selection (use all). Dense computation. Uses extreme compression for dense compute. Further reduces KV size.

CSA2 : Configurable compression. Full / Reuse / Reindex multi-mode. Sparse. Cross-layer reuse of index and KV. Reduces redundant computation, improves throughput.

LSA : Inherits CSA. Additional Memory Indexer (every τ steps). Sparse. GPU↔CPU dynamic recall. Decouples GPU memory usage .

6. Static Engram Decoupling vs Dynamic Cross-layer CSA2 Sharing

Both static decoupling and dynamic reuse are important across layers and parallelism.

Engram : Decouples static memory (N-gram) from computation for reuse.

CSA2 : Cross-layer CSA achieves compute sharing via ReIndex and ReUse.

7. Residual vs Hyper-Connection vs Attention Residual vs Depth Attention

Cross-layer connections address gradient vanishing; learnability and efficiency matter.

Standard Residual : Single-stream simple addition. No multi-stream. No constraints. Direct addition. Mitigates gradient vanishing (basic).

Hyper-Connection (HC) : Multi-stream residual. Yes multi-stream. No constraints. Learnable linear mapping. Enhances information flow.

mHC : Multi-stream residual. Yes. Manifold constraints. Constrained linear mapping. Enhances flow while ensuring training stability.

Attention Residual : Uses attention-weighted aggregation. Can be viewed as multi-stream. Constrained by attention weights. Attention aggregates historical layers . Allows current layer to selectively integrate all previous layers.

Depth-attention : Moves cross-layer residual/hidden-state inside attention via Value mixing.

8. Layer-wise Consistency vs Alternation vs Hybrid

Choosing between per-layer attention differences or a universal module remains a trade-off.

DeepSeek v4 : CSA and HCA alternate.

DeepSeek V4.1 : Universal module CSA2 with Full, Reuse, ReIndex configs.

Summary & Future Outlook

Amid the MoE explosion (referenced article: "Skills: The Third Great Compromise in the LLM Era"), attention is undergoing end-to-end optimization. Anthropic's prospectus reveals they are letting agents develop models themselves; future iterations may feed all open-source papers to Claude Code for self-testing new architectures. Anthropic defines a "Compute Multiplier" metric for iteration efficiency gains, marking the RSI (Recursive Self-Improvement) era.

Domestic papers become RSI input; within a week RSI validates feasibility at small scale, within half a month massive compute yields 4× efficiency improvement.

References

https://sebastianraschka.com/llm-architecture-gallery/images/hero/architecture-gallery-hero-backup.webp

https://www.anthropic.com/institute/recursive-self-improvement

The Evolution of Attention in Large Language Models 2609.39661v1.pdf

Long-Context Attention in Open-Source Large Language Models.pdf