How Bayesian Linear Regression Reveals Uncertainty in Model Parameters

Model

Given independent training samples, Bayesian linear regression uses the multivariate linear model y = Xw + ε , where w are the weight coefficients and ε are residuals assumed i.i.d. Gaussian. The residual variance follows an inverse‑Gamma distribution, requiring at least two hyper‑parameters (mean of the Gaussian and the inverse‑Gamma parameters). The framework can be extended to generalized linear models.

Solution

Using Bayes' theorem, the posterior of the weights is derived from the likelihood (determined by the linear model) and the prior. The likelihood is Gaussian when residuals are Gaussian. The marginal likelihood (model evidence) depends only on the data. A common prior is a zero‑mean Gaussian, but other priors such as uniform (uninformative) are also possible.

Maximum A Posteriori (MAP) Estimation

MAP treats the posterior mode as the estimate, turning the problem into an optimization similar to maximum likelihood. With a zero‑mean Gaussian prior, MAP coincides with ridge regression; with a Laplace prior, it corresponds to LASSO, yielding sparse solutions. MAP provides a point estimate without confidence intervals and supports fast computation.

Numerical Methods

General Bayesian inference methods, especially Markov Chain Monte Carlo (MCMC), apply to Bayesian linear regression. Gibbs sampling iteratively draws each weight from its conditional distribution given the current values of the other weights and the residual variance. The algorithm repeats until convergence.

Prediction

For MAP solutions, predictions are obtained by plugging the estimated weights into the linear model, yielding point predictions. For conjugate priors or MCMC solutions, predictions are obtained by marginalizing over the posterior of the weights, producing predictive distributions with credible intervals.

Model Validation

The marginal likelihood quantifies how well the model and prior explain the data and can be used to compute Bayes factors for model comparison. It integrates over all weight configurations, reflecting model complexity and prior assumptions.

Properties

Robustness: With a Gaussian prior, MAP equals ridge regression; with a Laplace prior, it yields LASSO‑like sparsity, balancing empirical risk and model complexity.

Bayesian advantages: Real‑time prior updating, no asymptotic assumptions, likelihood principle compliance, and predictive intervals. Unlike ordinary least squares, Bayesian linear regression works well even when the number of features exceeds the number of observations.

Relation to Gaussian Process Regression: Bayesian linear regression is a weight‑space view of GPR; with an identity mapping function, GPR reduces to Bayesian linear regression.