Why PCA Can Be Seen as Linear Regression: The Minimum Square Error Perspective

PCA Minimum Square Error Theory

After previous discussions on PCA from the maximum‑variance viewpoint, we now examine PCA from the minimum‑square‑error perspective, showing its equivalence to a linear regression problem.

Scene Description

We revisit the idea of exploring the universe of data, moving from dimensionality concepts to machine learning, and link PCA to dimensionality reduction.

Problem Statement

Since PCA seeks the optimal projection direction (a line), which coincides with the goal of linear regression, can we define PCA’s objective as a regression problem and solve it accordingly?

Background Knowledge

Linear algebra.

Solution and Analysis

Consider a two‑dimensional sample set. The maximum‑variance approach finds a line that maximizes the variance of projected points. This is analogous to linear regression, which finds a line that best fits the points by minimizing the sum of squared distances.

Extending to a d‑dimensional space, the goal becomes finding a d‑dimensional hyperplane that minimizes the sum of squared distances from the data points to the hyperplane. For the one‑dimensional case, the hyperplane reduces to a line, and the objective is to minimize the sum of squared distances from all points to this line.

By expanding the projection vector and separating terms, the minimization reduces to a sum over k of … (formula). The cross‑terms vanish because the projection vectors are orthogonal for i≠j, leaving only d terms.

When we solve for the d basis vectors ω₁, ω₂, …, ω_d of the projection matrix W, we obtain the same solution as the maximum‑variance method. For d=1, the optimal line ω coincides with the eigenvector corresponding to the largest eigenvalue of the covariance matrix, differing only by a scaling factor and a constant bias.

Summary and Extensions

From the minimum‑square‑error viewpoint we have derived PCA’s objective function and solution, which turn out to be equivalent to those obtained from the maximum‑variance viewpoint, confirming that both approaches lead to the same optimal projection.