Unlocking Multiple Linear Regression: Theory, Estimation, and Prediction

Multiple Linear Regression Model

Multiple regression analysis is a statistical method for studying the relationships among random variables. By analyzing observed data, a quantitative relationship (regression equation) between one dependent variable and a set of independent variables is established; after statistical testing confirms significance, the model can be used for prediction and control.

Least‑Squares Estimation of Regression Coefficients

Given a sample of observations, the parameters are estimated by the ordinary least‑squares method, which selects coefficient estimates that minimize the sum of squared residuals. This leads to the normal equations, which can be expressed in matrix form. When the design matrix has full column rank, the normal matrix is invertible and the coefficient estimates are obtained by solving the linear system.

The fitted values are computed by substituting the estimated coefficients into the regression equation, and the residuals (differences between observed and fitted values) represent the estimated random errors. The residual sum of squares (RSS) quantifies the overall lack of fit.

Testing Regression Coefficients

Statistical tests are required to verify whether the assumed linear relationship holds and whether each independent variable significantly influences the dependent variable. The total sum of squares is decomposed into regression sum of squares (explained variation) and residual sum of squares (unexplained variation). Hypothesis tests based on F‑statistics or t‑statistics assess the overall model and individual coefficients.

If the null hypothesis that all coefficients are zero is rejected, the model is considered significant. Further tests can identify which specific coefficients differ from zero, leading to model refinement by removing insignificant variables.

Prediction with the Regression Equation

For a given set of predictor values, the regression equation provides a point prediction of the response. Interval estimates can also be constructed: a confidence interval for the mean response and a prediction interval for a new observation. Approximate 95% prediction intervals are derived from the estimated variance and the leverage of the predictor values.

References

司守奎，孙玺菁. Python数学实验与建模.