Can DeepSeek’s mHC Architecture Break ResNet’s Decade-Long Dominance?

Challenging a Decade of ResNet Dominance

ResNet introduced in 2016 used a simple shortcut (residual) connection, expressed as “output of next layer = input of previous layer + new learned features”. This design became the foundation of Transformers, GPT, LLaMA and other large models. However, as models grew to billions of parameters, researchers found that plain residual connections limited further scaling, leading to the exploration of “Hyper‑Connections” (HC) that widen the residual pathway.

Problems with Naïve Hyper‑Connections

While HC can improve performance, the paper notes that it breaks the identity‑mapping property of residual connections, causing gradient instability (vanishing or exploding) during training of very large models. The authors show that a standard HC architecture causes loss to spike around 12k steps, resulting in training collapse.

DeepSeek’s Solution: Manifold‑Constrained Hyper‑Connections (mHC)

DeepSeek proposes mHC, which constrains the hyper‑connection matrices to be doubly stochastic using the Sinkhorn‑Knopp algorithm. This enforces energy conservation—preventing signal amplification or attenuation—and enables full information fusion across parallel “lanes”. The core idea is to project the unordered HC matrix onto the Birkhoff polytope, preserving an ordered mathematical space.

The paper highlights two key benefits:

Energy Conservation : ensures signal magnitude remains stable across layers.

Full Fusion : allows different data streams to exchange information, enhancing model expressiveness.

Engineering Optimizations to Keep Overhead Low

To offset the additional matrix projection cost, DeepSeek applies several system‑level optimizations:

Kernel Fusion : merges multiple compute steps to reduce GPU memory traffic.

Recomputing : discards intermediate activations during forward pass and recomputes them during back‑propagation to save memory.

DualPipe Communication Overlap : overlaps data transfer with computation, keeping GPUs fully utilized.

Experimental Results

On a 27 B parameter model, mHC increases training time by only 6.7 % compared with the baseline, while delivering substantial stability improvements and higher downstream performance. Benchmarks such as BBH, DROP and GSM8K show gains of 2.1 % and 2.3 % respectively over both the baseline and standard HC.

Heat‑map visualizations of the projection matrices illustrate that mHC produces orderly signal propagation, in contrast to the chaotic patterns observed with naïve HC.

Implications

The authors argue that mHC represents a low‑level innovation that challenges the long‑standing residual connection paradigm and paves the way for training even larger and deeper models. By open‑sourcing the technique, DeepSeek continues its commitment to community‑driven advancement.

https://arxiv.org/pdf/2512.24880