Sep 27, 2026 · Artificial Intelligence

How DeepSeek mHC Uses Its Residual Streams: Selective Routing and Near-Identity Mixing

Analysis of DeepSeek-V4-Flash's mHC architecture shows its four residual streams are not uniformly utilized; read/write weights concentrate on ~2 streams per layer, and deep-layer residual mixing matrices approach identity. Interventions removing weakest paths or replacing deep mixing with identity cause minimal performance drops (≤0.38% avg score), suggesting simplification opportunities.

DeepSeek-V4-FlashHyper-Connectionsarchitecture simplification

0 likes · 12 min read

How DeepSeek mHC Uses Its Residual Streams: Selective Routing and Near-Identity Mixing

Machine Learning Algorithms & Natural Language Processing

Jul 27, 2026 · Artificial Intelligence

Scaling Residual Streams Efficiently: From DeepSeek mHC to xHC’s 16‑Stream Expansion

The blog details how xHC expands language‑model residual streams to 16, achieving nearly double the gain of DeepSeek mHC on 18B and 28B MoE models, and explains the design of Temporal Feature Augmentation and Sparse Write that make large‑N scaling both effective and affordable.

Hyper-ConnectionsLarge Language ModelsMixture of Experts

0 likes · 24 min read

Scaling Residual Streams Efficiently: From DeepSeek mHC to xHC’s 16‑Stream Expansion