Why Linear Regression Matters: Theory, Python Implementation, and Boston Housing Prediction

Linear Regression Overview

Linear regression models the relationship between one or more explanatory variables (features) and a continuous target variable by fitting a linear function that minimizes the sum of squared errors.

Simple (Univariate) Linear Regression

Given paired observations (x_i, y_i), the model assumes y = a·x + b. The loss function is the ordinary least‑squares (OLS) error: L(a,b) = \sum_{i=1}^{n} (y_i - (a·x_i + b))^2 Setting the partial derivatives of L with respect to a and b to zero yields closed‑form solutions:

a = \frac{\sum (x_i - \bar{x})(y_i - \bar{y})}{\sum (x_i - \bar{x})^2},\quad b = \bar{y} - a·\bar{x}

Typical Python implementation defines two methods: def fit(X, y): – compute a and b from the training data. def predict(X): – apply the learned line to new x values.

Vectorized NumPy operations replace explicit loops, improving readability and enabling GPU acceleration.

Multivariate Linear Regression

When multiple features x = (x_1, x_2, …, x_p) are available, the model becomes

y = \theta^T x = \theta_0 + \theta_1 x_1 + … + \theta_p x_p

Stacking all samples into a design matrix X \in \mathbb{R}^{n\times(p+1)} (with a column of ones for the intercept) and the target vector y \in \mathbb{R}^n, the predictions are \hat y = X\theta. Minimizing the same squared‑error loss leads to the normal equation: \theta = (X^T X)^{-1} X^T y The normal equation provides an exact solution when X^T X is invertible, but for large or ill‑conditioned datasets iterative solvers such as gradient descent are preferred.

Practical Example: Boston Housing Price Prediction

The Boston housing dataset (included with scikit‑learn) contains 506 samples, each with 12 features describing neighborhood characteristics and the median house price.

Key steps:

Load the data using sklearn.datasets.load_boston() (or the updated fetch_openml interface).

Separate the feature matrix X and target vector y.

Optionally standardize features to improve numerical stability.

Fit a multivariate linear regression model using the normal equation or a library such as sklearn.linear_model.LinearRegression.

Inspect the learned coefficients θ to identify which features have positive or negative influence on price.

Interpretability is a major advantage of linear regression: each coefficient directly quantifies the expected change in the target per unit change in the corresponding feature, holding other features constant.

By applying the trained model, practitioners can predict house prices for new neighborhoods and understand the relative importance of factors such as nitric oxide concentration, distance to employment centers, and crime rate.