How Attention Residuals Boost Transformer Efficiency and Scale

Attention Residuals Overview

Attention Residuals replace the uniform additive residual connection in deep Transformers with a softmax‑based attention aggregation. Each layer owns a learnable pseudo‑query vector that scores the outputs of all preceding layers, allowing the layer to selectively attend to any earlier representation. The aggregated result is normalized with RMSNorm to prevent any single layer from dominating the attention distribution.

Full Attention Residual (Full AttnRes)

In the Full AttnRes configuration every transformer layer is equipped with an independent query vector. During the forward pass the layer computes a similarity score between its query and the outputs of all previous layers, applies a softmax to obtain attention weights, and then forms a weighted sum of those outputs. RMSNorm is applied to the summed representation before it is passed to the next sub‑layer. The per‑token computational overhead is modest because the number of layers is far smaller than the sequence length.

Block Attention Residual (Block AttnRes)

Block AttnRes groups consecutive layers into fixed‑size blocks. Within a block the traditional residual addition is retained, producing a single block‑level representation. Across block boundaries the full attention residual mechanism is applied only to these block representations together with the original token embeddings. This reduces memory consumption and inter‑node communication while preserving the ability to attend across distant depths.

Distributed Training Optimizations

To scale Attention Residuals to billions of parameters, a two‑phase computation strategy is introduced. Phase 1 performs parallel attention across blocks using the pre‑learned query vectors, which are independent of the current forward computation. Phase 2 serially processes intra‑block accumulation and merges the results with an online softmax. A cross‑stage cache stores unchanged block outputs so that only newly computed increments are transmitted between pipeline stages, dramatically lowering communication volume and hiding latency.

Scaling‑law Experiments

Models of varying size—including MoE‑augmented Transformers with an 8 k token context window and cosine learning‑rate scheduling—were trained under identical compute budgets. Both Full AttnRes and Block AttnRes consistently achieved lower validation loss than the baseline residual architecture, and the gap widened with model scale. Block AttnRes matched Full AttnRes performance while using roughly 1.25× the compute of the baseline.

Ablation Studies

Replacing static query vectors with dynamic, input‑dependent ones yields a small accuracy gain but adds extra linear projections that increase inference memory usage. Removing RMSNorm or swapping the softmax activation for sigmoid leads to noticeable performance degradation, confirming the importance of competitive softmax normalization for sharp attention selection.

Impact on Model Topology

Heat‑map sweeps over depth‑width configurations show that traditional residual networks perform best at a depth‑width ratio of ≈60, whereas Attention Residuals shift the optimum to ≈45, encouraging deeper‑narrower backbones. Visualizations of attention weights reveal strong diagonal dominance with occasional long‑range jumps, indicating learned cross‑block connections.

Conclusions

Attention Residuals provide a principled solution to the memory‑dilution problem in deep Transformers, delivering consistent accuracy improvements with modest compute and memory overhead. The engineering optimizations make the design practical for large‑scale distributed training, positioning Attention Residuals as a strong candidate for next‑generation large language models.

Reference implementation: https://github.com/MoonshotAI/Attention-Residuals Pre‑print:

https://arxiv.org/pdf/2603.15031