Hybrid Attention: Why Kimi and DeepSeek Now Share a Model Architecture

From Full Attention to Hybrid Architectures

The article opens with a striking observation: a single model architecture diagram now includes both Kimi and DeepSeek. Specifically, GLM-5.3-Flash's 45-layer configuration uses 34 layers of Kimi's KDA (linear attention) and 11 layers of DeepSeek's KPool-DSA (sparse attention). This reflects a broader trend: the role once reserved for global attention in hybrid architectures is increasingly being taken over by sparse attention.

Three Categories of Attention

The author categorizes attention mechanisms into three types:

Global (Full) Attention — reads the complete token history, preserving maximum information but with the highest compute and KV cache cost.

Linear Attention — compresses history into a compact state to reduce long-sequence compute cost.

Sparse Attention — retains token-level history but selects only a subset for computation.

Efficient attention research has largely evolved along the latter two directions.

The Hybrid Compromise

Classic Transformers used full attention, but its quadratic cost led to hybrid designs: efficient attention handles bulk computation while a few global attention layers preserve direct access to full context. However, in 2024 models like Qwen3.8-Flash-Next replaced that global layer with Qwen Sparse Attention (QSA), keeping the 3:1 Gated DeltaNet to Attention ratio but changing the "fallback" mechanism.

MiniMax's Architectural Journey

Phase 1: Betting on Linear Attention (Text-01)

In late 2023, MiniMax researcher Zhong Yiran (who had scaled linear attention to 15B) joined MiniMax after founder Yan Junjie agreed to back the unproven-at-scale approach. Zhong estimated 99% success probability; Yan estimated ~50%. MiniMax committed >80% of R&D resources to this bet.

Early 15B linear-only experiments looked promising, but scaling revealed a critical flaw: linear attention's compression hurts retrieval tasks (e.g., needle-in-haystack). The analogy: reading a book while taking compressed notes — efficient for gist, poor for exact page-37 recall.

Phase 2: 7:1 Hybrid with Full Attention (MiniMax-01)

MiniMax-01 adopted 7 layers of Lightning Attention + 1 layer of SoftMax Attention. The ratio came from ~3,700 pre-training experiments testing scaling laws. The full-attention layer acted as a "safety net" for unknown capability gaps. The 456B model launched in Jan 2025 with 4M-token context.

Phase 3: Back to Full Attention (M2)

Further scaling exposed multi-hop reasoning degradation in hybrid models. While small-scale fixes closed the gap, the team realized unknown capability losses cannot be exhaustively pre-tested . Benchmarks used for Text-01 didn't reveal the multi-hop gap. M2 therefore reverted to full attention, accepting higher cost for predictable capability boundaries.

Phase 4: Sparse Attention for Agent Workloads (M3)

Agent workloads changed the cost equation: context becomes a growing work history (search results, code, tool outputs, errors) accumulating over dozens of turns. Full attention's compute and KV cache grow with every turn. M3 introduced MiniMax Sparse Attention (MSA), which selects only relevant context portions for attention. At 1M tokens, M3 achieves ~1/20 the per-token compute of its predecessor. MiniMax labeled "Context Scaling" as the core challenge for complex agents.

Convergence: Linear + Sparse in One Model

By 2026, multiple models combine linear and sparse attention:

MiniCPM-SALA (Feb 2026): Lightning Attention + Sparse Attention

Qwen3.8-Flash-Next (Aug 2026): 3-layer Gated DeltaNet + 1-layer Qwen Sparse Attention

GLM-5.3-Flash: 34-layer KDA (Kimi) + 11-layer KPool-DSA (DeepSeek)

The combination works because they solve complementary problems: linear attention cheaply maintains long-term state; sparse attention retrieves specific details from token-level history when needed.

Technology Diffusion Across Company Boundaries

Open-source infrastructure (e.g., Yang Songlin's FLA library for linear attention) and talent mobility have made attention mechanisms reusable components rather than proprietary "routes." Yang predicted linear+sparse fusion in late 2025; within a year it appeared in production models.

Conclusion: Routes Are Now Continuous Judgment Calls

The article argues that "technical route" labels (MiniMax=Linear, DeepSeek=Sparse, Kimi=KDA) are expiring. The real differentiator is when to adopt a mechanism, how to combine it, and what trade-offs to accept among capability, cost, and engineering risk . MiniMax's pivots — Linear → Hybrid+Full → Full → Sparse — were each driven by the constraint that hit hardest at that stage: context cost → unknown capability risk → agent context scaling. GLM-5.3-Flash's KDA+DSA coexistence records not a winner but a snapshot: as techniques become verified, absorbable, and recomposable, a "route" becomes a team's ongoing judgment of where the next wall will appear.

Technology diffuses and recombines. What's hard to copy is a team's ability to anticipate the next bottleneck.