Lao Guo's Learning Space

Oct 25, 2026 · Artificial Intelligence

Attention Algorithms 2017-2026: MHA, GQA, MLA, NSA Selection Guide

This comprehensive survey covers attention mechanisms from 2017's Scaled Dot-Product Attention to 2026's NSA and FlashAttention-4, organizing them into four optimization tracks—KV cache compression, exact attention acceleration, long-context extension, and frontier alternative architectures—with concrete benchmarks, trade-offs, and practical selection criteria for each use case.