Scaling Residual Streams Efficiently: From DeepSeek mHC to xHC’s 16‑Stream Expansion

Introduction

We recently released xHC (Expanded Hyper‑Connections), which for the first time scales the residual stream of a language model to 16 streams efficiently. In pre‑training experiments on 18B and 28B MoE LLMs, xHC builds on DeepSeek Manifold‑constrained Hyper‑Connections (mHC) and almost doubles the performance gain.

Reproducing mHC

Starting from the mHC release, we first reproduced its results. The original paper showed a clear advantage over the vanilla residual connection, with average scores rising from 40.6 to 44.8 points on an 18B MoE backbone. Specific benchmarks (MMLU, ARC‑Challenge, GSM8K, C3) improved by 5.8, 10.6, 6.8 and 5.6 points respectively, confirming that residual‑stream expansion is a promising direction.

Paper link: https://arxiv.org/pdf/2607.14530
Github link: https://github.com/aHapBean/xHC

Why Scale the Number of Streams?

Most published HC/mHC experiments fix the expansion rate N at 4. While increasing from 1 to 4 yields noticeable gains, further growth to 8 shows diminishing returns. This raised the question: must N stay at 4, or can we continue scaling it?

Exploring Larger N

We kept all settings identical and only increased the number of residual streams. In a 2.5B MoE test, expanding N from 4 to 8 reduced validation loss by only 0.006 while FLOPs grew by 32 %, indicating a poor cost‑benefit trade‑off.

Designing xHC

Two problems guided the design:

Why does the benefit plateau after a certain N?

Why does the computational cost rise so sharply with more streams?

To address them we introduced:

Temporal Feature Augmentation (TFA) : a causal depthwise convolution that extracts multi‑granularity temporal features from neighboring tokens, providing additional write‑back components without extra sub‑layer computation.

Sparse Write : a dense‑read / sparse‑write architecture where only a subset of streams (e.g., 4 out of 16) are updated per layer, dramatically reducing the cubic projection cost.

Component Ablations

On a 10B MoE model we performed controlled ablations:

Expanding mHC directly to N=16 reduced loss from 2.004 to 1.998 but increased FLOPs from 0.6 % to 18.8 %.

Adding TFA lowered loss further to 1.984, confirming that richer write‑back information yields gains.

Replacing the dense residual update with xHC’s sparse write dropped loss to 1.983 while cutting extra FLOPs from 20.1 % to 3.3 %.

These results show that TFA solves the information‑supply bottleneck, while Sparse Write controls the computational overhead.

Main Results

Under equal FLOPs, xHC outperforms Vanilla and mHC on both 18B and 28B MoE models:

18B: average scores 40.6 (Vanilla) → 44.8 (mHC) → 48.8 (xHC), a 4.0‑point gain over mHC.

28B: average scores 47.8 → 50.5 → 53.6, a 3.1‑point gain over mHC.

Scaling Law Evaluation

We trained models with varying compute budgets and fitted scaling curves. Across the tested range, xHC consistently achieved the best scaling curve; Vanilla and mHC required 1.50× and 1.19× more FLOPs respectively to reach the same loss.

Cost Analysis and xHC‑Flash

Even with Sparse Write, memory‑access (I/O) remains the dominant cost. xHC‑Flash reduces theoretical memory traffic from 73.5C to 40C, comparable to mHC’s 34C, by further optimizing the dense‑read correction.

Conclusion

xHC demonstrates that residual‑stream expansion can be a true scaling axis when combined with Temporal Feature Augmentation and Sparse Write. The method yields substantial performance gains on large‑scale MoE LLMs while keeping additional compute and memory overhead manageable. The full implementation will be open‑sourced on GitHub.