Master Linear, Weighted, and Ridge Regression: Theory, Code, and Evaluation

Regression

Machine learning algorithms are divided by the type of target variable: categorical data leads to classification, while continuous data leads to regression.

Meaning

The term “regression” originates from Francis Galton’s 1877 work predicting pea seed sizes from parent measurements, observing that offspring tend to revert toward the mean.

Types

Regression methods include linear regression and nonlinear regression. Linear regression fits a straight line; nonlinear regression handles feature interactions, with logistic regression being a special case used for classification.

Linear Regression

Principle Overview

Linear regression models the relationship between two variables by deriving a regression equation that predicts the outcome y from input x.

For example, using housing price data:

The equation shows a linear relationship between price, area, and orientation, with coefficients 0.7 and 0.19.

Representing features as matrix X (m×n) and coefficients as θ (n×1), predictions are Y = Xθ. The goal is to find θ that minimizes the squared error, leading to the normal equation:

Deriving θ involves matrix inversion, which requires XᵀX to be invertible.

Application

The following Python code implements the linear regression algorithm from scratch:

def standRegres(xArr, yArr):
    xMat = mat(xArr)
    yMat = mat(yArr).T
    xTx = xMat.T * xMat  # check if inverse exists
    if linalg.det(xTx) == 0.0:
        print "This matrix is singular, cannot do inverse"
        return
    ws = xTx.I * (xMat.T * yMat)
    return ws

After obtaining the regression coefficients, predictions are made by multiplying the feature values.

Python’s sklearn library provides a ready‑made implementation:

from sklearn import linear_model
regr = linear_model.LinearRegression()
regr.fit(x, mydata)
predict_outcome = regr.predict(x)

Model Evaluation

Model quality can be assessed by comparing predicted and actual values, e.g., using the correlation coefficient:

A higher correlation indicates a better fit.

Locally Weighted Linear Regression

To address under‑fitting of ordinary linear regression, locally weighted linear regression assigns higher weights to points near the prediction target, using a kernel function (commonly Gaussian):

The bandwidth parameter k controls weight decay; smaller k yields faster decay, emphasizing nearby points.

Experiments with k = 1, 0.01, and 0.003 show that k = 0.01 provides the best balance between bias and variance.

Ridge Regression

When XᵀX is singular (e.g., due to multicollinearity), ordinary linear regression fails. Ridge regression adds a regularization term λI to make the matrix invertible:

The objective becomes minimizing the squared error plus λ times the squared norm of θ, reducing coefficient magnitude and handling collinearity.

The ridge trace plot shows how coefficients change with λ:

Conclusion

We reviewed three regression models: ordinary linear regression, locally weighted linear regression (to mitigate under‑fitting), and ridge regression (to address multicollinearity). Selecting the appropriate model depends on the data characteristics and the specific problem at hand.