Why Residual Connections Keep Deep Neural Networks Stable

Why Do We Need Residual Connections?

In large language models such as Transformers, information passes through many modules (e.g., self‑attention, feed‑forward networks). As the number of layers grows, two major issues arise: network degradation —accuracy can drop despite deeper models, and gradient‑propagation difficulty —training signals struggle to reach early layers, slowing or preventing learning.

How Do Residual Connections Work?

A residual (shortcut) connection provides a direct path that adds the original input x to the output of a processing block F(x). The combined result is then passed to the next layer: output = F(x) + x Here x is the input vector, and F(x) is the transformation performed by the current module (e.g., a self‑attention block).

This can be visualized as a production line where each station not only processes the material but also forwards the raw material unchanged to the next station, ensuring the original information is never lost.

Why Do Residual Connections Enable Deep Training?

The addition works because the tensors have the same shape (identical dimensionality). Designers align the dimensions of each module’s output with its input, similar to matching conveyor‑belt widths in a factory. When F(x) performs poorly, the shortcut allows the layer to behave like an identity mapping, preserving the input.

Consequently, gradients can travel back through the shortcut path directly, reducing attenuation and making back‑propagation more stable. This stability lets models be stacked to hundreds of layers, which is a key factor behind the success of architectures such as Transformer, GPT, and BERT.

x = [1.0, 2.5, -0.3]
F(x) = [0.5, -1.5, 0.2]

Both vectors have the same dimensionality, so they can be added element‑wise, demonstrating the shape‑matching requirement.

In summary, residual connections act as simple additive shortcuts that preserve the original signal, ensure gradient flow, and allow deep neural networks to be trained reliably, which is fundamental to modern large‑scale language models.