Master Linear Regression: Concepts, Math, and Python Implementation

Linear regression is a cornerstone algorithm in machine learning, providing a simple yet powerful way to model linear relationships between variables for tasks such as house‑price prediction, stock analysis, biomedical data, and industrial optimization.

Basic Concepts

Definition and Formula

Linear regression builds a statistical model that assumes a linear relationship between one or more independent variables (features) and a dependent variable (target). When there is a single feature, it is called simple (univariate) regression; with multiple features, it becomes multiple (multivariate) regression.

Simple Linear Regression

The model is expressed as y = w·x + b + ε, where y is the target, x the feature, w the slope, b the intercept, and ε the error term.

Multiple Linear Regression

For p features, the model generalises to y = X·β + b + ε, where X is the design matrix, β the vector of coefficients, and ε the error term.

Model Assumptions

The key assumption is that the target is a linear combination of the features plus an error term that follows a normal distribution with mean zero. This enables statistical inference and the use of ordinary least squares.

Loss Functions and Optimization

Residual Sum of Squares (RSS)

RSS = \(\sum_{i=1}^{n}(y_i - \hat{y}_i)^2\) measures the total squared error between predictions and observations.

Mean Squared Error (MSE)

MSE = RSS / n, providing an average error metric that is comparable across datasets.

Ordinary Least Squares (OLS)

OLS finds the coefficient vector that minimises RSS. For the multivariate case, the solution is obtained by solving the normal equation (XᵀX)β = Xᵀy, assuming XᵀX is invertible.

Gradient Descent

When the normal equation is computationally expensive, gradient descent iteratively updates the coefficients:

Initialize β (e.g., zeros or random values).

Compute the gradient of the loss with respect to β.

Update β ← β - η·gradient, where η is the learning rate.

Repeat steps 2‑3 until convergence or a maximum number of iterations.

Variants include Batch Gradient Descent (using the whole dataset) and Stochastic/ Mini‑Batch Gradient Descent (using one or a few samples per iteration), each with trade‑offs in speed and stability.

Python Implementation with scikit‑learn

Install required packages: pip install numpy pandas scikit-learn Load and preprocess data:

import pandas as pd

data = pd.read_csv('house_prices.csv')
X = data[['area', 'rooms', 'age']]
y = data['price']

Split into training and test sets:

from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)

Train the model:

from sklearn.linear_model import LinearRegression
model = LinearRegression()
model.fit(X_train, y_train)

Make predictions and evaluate:

y_pred = model.predict(X_test)

from sklearn.metrics import mean_squared_error, mean_absolute_error, r2_score
mse = mean_squared_error(y_test, y_pred)
rmse = mean_squared_error(y_test, y_pred, squared=False)
mae = mean_absolute_error(y_test, y_pred)
r2 = r2_score(y_test, y_pred)

print(f"MSE: {mse}")
print(f"RMSE: {rmse}")
print(f"MAE: {mae}")
print(f"R²: {r2}")

Case Studies

House‑Price Prediction

Explore data with matplotlib and seaborn to confirm strong positive correlation between area and price.

Train a linear model; coefficients reveal that area has the largest positive impact, while age has a negative impact.

Evaluate using MSE, RMSE, MAE, and R² to assess fit.

Sales Forecasting

Preprocess sales data, handling missing values and outliers.

Build a regression model with advertising spend, promotion count, and price as features.

Assess performance with the same metrics; use predictions for production planning.

Advantages and Limitations

Advantages

Strong interpretability – coefficients directly indicate feature influence.

High computational efficiency, especially with OLS.

Simple formulation makes it an ideal entry point for beginners.

Extensible to polynomial features, regularisation, and ensemble methods.

Limitations

Assumes linearity; fails on inherently non‑linear relationships.

Sensitive to outliers due to squared error minimisation.

Prone to over‑fitting on small datasets.

Vulnerable to multicollinearity among features.

Conclusion and Outlook

Linear regression remains a fundamental, interpretable, and efficient algorithm for a wide range of predictive tasks. While its simplicity is a strength, practitioners must be aware of its assumptions and limitations, and may combine it with regularisation, feature engineering, or more complex models as data complexity grows.