Why Layer Normalization Stabilizes Transformers: A Deep Dive

Background

The piece is part of a series that dissects the long‑form article "Understanding LLMs from Scratch Using Middle School Math" and focuses on the ninth chapter, which introduces layer normalization as a crucial stabilizer for neural networks.

Standard Deviation and Z‑score

Standard deviation measures how spread a set of numbers is. To compute it:

Calculate each number’s deviation from the mean.

Square each deviation.

Average the squared deviations (variance).

Take the square root of the variance (standard deviation).

Example: scores 20, 50, 60, 80, 90 have mean 60, deviations –40, –10, 0, 20, 30; variance = (1600+100+0+400+900)/5 = 600; standard deviation ≈ 24.49.

The Z‑score (standard score) of a value is (value – mean) / standard deviation, converting raw numbers into a common scale.

Why Layer Normalization?

In deep models such as Transformers, neuron outputs can vary wildly across layers, making training unstable or causing convergence failure. Layer normalization applies the Z‑score idea to each sample’s activations, ensuring a uniform scale and improving stability, especially for sequence data and small batches.

Layer Normalization Mechanics

The normalized output z_i is transformed by learnable parameters γ (scale) and β (bias):

y_i = γ_i * z_i + β_i

γ_i * z_i

: scaling factor that can enlarge (γ > 1) or shrink (γ < 1) the normalized value, allowing important neurons to retain influence. + β_i: bias that shifts the entire value up or down, preserving useful positional information.

Scaling and Bias Example

Analogous to exam scores, γ can amplify a subject’s importance (e.g., γ = 2 for math), while β can shift the overall standardized score to a more convenient range, similar to moving a Z‑score baseline.

Key Benefits

Stabilizes training by keeping activations centered and scaled.

Retains the model’s ability to differentiate important signal variations through learnable γ and β.

Operates per sample, making it suitable for Transformers and other sequence models.

Practical Note

In the classic Transformer architecture, LayerNorm is placed after the residual connection; many modern variants apply it before the residual connection, but the combination of residual pathways and LayerNorm consistently improves deep model training.