How Manifold-Constrained Hyper-Connections Boost Large Model Training Efficiency

Background: Residual Connections and Hyper‑Connections

Since ResNet, the identity‑mapping residual connection has been the standard because it preserves signal mean and prevents gradient explosion or vanishing. Hyper‑Connections (HC) expand a single residual stream into n parallel streams, dramatically increasing performance but causing numerical instability and large memory‑bandwidth overhead.

mHC: Manifold‑Constrained Hyper‑Connections

mHC locks the free matrix of HC onto the Birkhoff polytope using a "double stochastic matrix". This constraint ensures:

Row and column sums equal 1, preserving signal mean.

Spectral norm ≤ 1, preventing gradient explosion.

Multiplicative closure, keeping deep networks stable.

With this constraint, the gradient gain drops from ~3000 to 1.6, allowing stable training of a 27 B‑parameter model.

Sinkhorn‑Knopp Projection

mHC obtains a doubly stochastic matrix by applying the Sinkhorn‑Knopp algorithm: 20 iterations of row‑ and column‑normalisation approximate the manifold with negligible computation.

Non‑Negative Input/Output Mapping

Sigmoid activation guarantees non‑negative inputs and outputs, avoiding cancellation between positive and negative signals.

Efficient Engineering Implementation (TileLang Fusion)

A custom TileLang kernel fuses RMSNorm, matrix multiplication, and the 20‑iteration Sinkhorn projection into three kernels, reducing memory traffic from (5n+1)C to (n+1)C and cutting bandwidth pressure by ~70%.

Segmented recomputation stores only the first‑layer input; deeper activations are recomputed on‑the‑fly, achieving near‑optimal pipeline stage alignment.

DualPipe communication overlap moves the most time‑consuming FFN‑post kernel to a high‑priority stream, overlapping computation and communication with >90% efficiency.

Experimental Results

Across eight downstream tasks, mHC consistently outperforms vanilla HC, achieving +2.1% BBH and +2.3% DROP improvements. Training a 27 B model shows:

Absolute loss reduction of 0.021 and a smooth gradient‑norm curve.

All eight benchmarks lead by an average of 2%.

Expanding n to 4 adds only 6.7% training time, with memory usage comparable to the baseline.

Figures illustrate standard residual connections, HC structure, the manifold‑constrained version, training stability curves, scaling curves, signal‑gain visualisation, and downstream‑task performance tables.

Conclusion

mHC delivers stable, fast, and scalable training for extremely large models with minimal overhead, demonstrating that manifold‑constrained hyper‑connections can reliably replace traditional residual paths in modern AI architectures.

https://arxiv.org/pdf/2512.24880