Skip to content
Five Cuts to Attention Optimization: MHA, MQA, GQA, FlashAttention Layer-by-Layer

Five Cuts to Attention Optimization: MHA, MQA, GQA, FlashAttention Layer-by-Layer

This article dissects MHA, MQA, GQA, and FlashAttention across five analytical layers, revealing how structural optimizations (MQA/GQA) reduce KV Cache memory by sharing keys/values, while FlashAttention tackles memory bandwidth via tiled computation and online softmax, and why modern LLMs combine GQA with FlashAttention for 2-4x speedups without accuracy loss.

Code Farming
Code Farming
Code Farming

First Cut: Separate Training and Inference — MHA's Pain Points Live in Different Places

The claim "O(N²) is slow" conflates two distinct phases. In training, a sequence of length N requires computing an N×N attention score matrix softmax(QKᵀ/√d)·V per layer, which must be stored for backpropagation. At N=4K this is 16M elements; at N=32K it balloons to 1B elements (~2 GB in FP16). However, this matrix is computed once and discarded.

Inference tells a different story. Autoregressive generation recomputes attention against all prior tokens for each new token, leading to O(N³) accumulation if done naively. KV Cache solves this by storing historical K and V so each new token only computes its own Q. But the cache itself becomes a memory hog.

Three intertwined pain points emerge: (1) N² compute complexity makes cost grow quadratically with sequence length; (2) KV Cache makes long-context memory explode; (3) Memory bandwidth prevents GPU compute utilization — Attention is memory-bound, not compute-bound.

Second Cut: The KV Cache Memory Bill — 7B at 32K Context Needs 17 GB

KV Cache size follows the formula:

2 (K and V) × B (batch) × N (seq len) × L (layers) × H (heads) × d_k (head dim) × 2 bytes (FP16)

Plugging in a 7B model (L=32, H=32, d_k=128) with batch=1, N=32K:

2 × 1 × 32000 × 32 × 32 × 128 × 2 ≈ 17 GB

Add 14 GB for model weights → 31 GB total, exceeding a 24 GB RTX 4090. This is the core long-context bottleneck. Moreover, GPU compute sits idle waiting for data from HBM (≈1.5 TB/s) while on-chip SRAM (≈19 TB/s) sits underutilized — the classic memory-bound problem.

Third Cut: MQA and GQA — Structural Sharing of K/V on a Spectrum

MQA (Multi-Query Attention) forces all heads to share a single K and V; only Q remains per-head. For H=32, KV Cache drops to 1/32 (17 GB → ~0.5 GB). Training pipeline barely changes. Cost: expressiveness collapses from 32 perspectives to one. Empirical drop 2-5%, severe on reasoning tasks (math, code).

GQA (Grouped-Query Attention) interpolates between MHA and MQA: partition H heads into G groups; heads within a group share K/V. G=H → MHA; G=1 → MQA. The sweet spot: H=32, G=8 yields 1/4 memory with <0.5% quality loss (per Meta's GQA paper). Hence Llama 2 70B, Llama 3, Qwen 2/3 adopt GQA. DeepSeek's MLA (Multi-Head Latent Attention) takes a more aggressive low-rank latent route — not merely a GQA upgrade.

Fourth Cut: FlashAttention — Implementation-Layer Tiling + Online Softmax

FlashAttention does not change the math; it changes the execution to attack the memory-bandwidth wall.

Hardware gap: HBM capacity large (A100 40/80 GB) but bandwidth ~1.5 TB/s. SRAM per SM tiny (192 KB) but bandwidth ~19 TB/s (10×+).

Standard implementation: writes full N×N matrices S=QKᵀ, P=softmax(S), O=PV to HBM — multiple round-trips dominate runtime.

FlashAttention: splits Q/K/V into blocks fitting in SRAM. Computes one output block at a time, accumulates directly into O, never materializing N×N intermediates in HBM. Challenge: softmax needs full row statistics. Solution: online softmax maintains running max and sum per row, updating incrementally block-by-block. Final result is mathematically equivalent to full softmax (only negligible FP rounding differences from block order).

Gains: (1) Memory from O(N²) → O(N); (2) HBM traffic from O(N²) → O(N²/M) where M is block size; (3) 2-4× wall-clock speedup; (4) No accuracy loss (not sparse/low-rank). Now at v3, default in vLLM, SGLang, TGI.

Fifth Cut: They Stack, Not Substitute — The High-Scoring Insight

MQA — Structural layer: K/V reduced to 1 copy. Targets memory. Quality impact: medium (2-5% drop).

GQA — Structural layer: K/V reduced to G copies. Targets memory. Quality impact: near-zero (<0.5% drop).

FlashAttention — Implementation layer: tiling + online softmax. Targets memory + bandwidth + speed. Quality impact: none (math-equivalent).

Key realization: MQA/GQA operate at the structural layer (how many K/V copies exist); FlashAttention operates at the implementation layer (how the same math is executed). They attack orthogonal bottlenecks and compose freely. Production standard: GQA + FlashAttention (Llama 3, Qwen 2, DeepSeek V3). GQA shrinks KV Cache to 1/4; FlashAttention accelerates compute 2-4×. Together a 7B model runs 32K context on a 4090 — unthinkable five years ago.

Interview Answer Skeleton

Lay out three pain points: KV Cache linear in heads & length → OOM; repeated HBM reads → bandwidth-bound; N² matrix → quadratic compute.

Position each solution: MQA brute-shares K/V (1/H memory, quality drop); GQA groups heads (near-MQA memory, near-MHA quality — sweet spot); FlashAttention changes implementation (O(N²)→O(N) memory, 2-4× speed, no quality loss).

Close with the stacking principle: structural + implementation layers target different knobs; real systems combine GQA + FlashAttention. Bonus: mention MLA as next-gen low-rank latent approach.

One-liner takeaway: Memorizing names scores zero; explaining "who lives at which layer and why they stack" is the foundation.

Original Source

Signed-in readers can open the original source through BestHub's protected redirect.

Sign in to view source
Republication Notice

This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactadmin@besthub.devand we will review it promptly.

Memory OptimizationFlashAttentionAttention MechanismLLM InferenceMHAKV CacheGQAMQA
Reader feedback

How this landed with the community

Sign in to like

Rate this article

Was this worth your time?

Sign in to rate
Discussion

0 Comments

Thoughtful readers leave field notes, pushback, and hard-won operational detail here.