Why Normal (Gaussian) Distributions Are Fundamental to Machine Learning

Introduction

Normal (Gaussian) distributions are a cornerstone of machine learning because many algorithms model data, noise, and parameters as Gaussian variables. The article assumes basic probability knowledge and focuses on the aspects most relevant to ML.

Central Limit Theorem (Review)

When we repeatedly sample the average of n independent random variables, the distribution of the sample mean approaches a normal distribution as n grows (about 30‑50 samples are often sufficient). This explains why normal patterns appear in real‑world data.

Multivariate Gaussian and Sampling

A multivariate normal distribution is denoted y \sim \mathcal{N}(\mu, \Sigma), where \Sigma is the covariance matrix and |\Sigma| its determinant. When \mu = 0 and \Sigma = I the distribution is called standard normal. To sample from a multivariate Gaussian we first draw X \sim \mathcal{N}(0, I) and then compute Y = \mu + A X, where A is obtained via a Cholesky decomposition of \Sigma, yielding a triangular matrix that reduces computational cost.

Key Properties of Gaussian Distributions

Product : The product of two Gaussian densities is proportional to another Gaussian density scaled by a factor s. The resulting mean and variance can be solved analytically (see the accompanying equations).

Sum : The sum of two independent Gaussian variables is again Gaussian, with mean equal to the sum of means and variance equal to the sum of variances.

Conditional Distribution

For a joint Gaussian vector (X, Y), the conditional distribution of X given Y = y is also Gaussian. The article derives the conditional mean and covariance analytically, showing the steps from the joint covariance matrix to the conditional parameters (see the series of equations).

Marginal Distribution

Integrating out variables from a joint Gaussian yields a marginal distribution that is itself Gaussian. The article illustrates this with a simple joint density and its marginal image.

Linear Transformation

If X and Y are independent Gaussian variables, applying a linear transformation A to X results in a new Gaussian variable Y = A X whose covariance is A \Sigma_X A^T. The derivation is shown with accompanying matrix equations.

Gaussian Priors and Bayesian Inference

In Bayesian inference, the marginal likelihood (denominator) is often intractable, but when both the prior p(θ) and the likelihood p(D|θ) are Gaussian, the posterior p(θ|D) remains Gaussian. The article walks through the algebra that collapses the product of prior and likelihood into a single Gaussian, which underlies Bayesian linear regression and Gaussian‑process models.