Understanding Support Vector Regression: Theory and Formulation

Support Vector Regression

Support Vector Regression (SVR) uses a training sample set as data objects, analyzes the quantitative relationship between input variables and a numeric output variable, and predicts the output for new observations.

Let \(x\) denote the input space, where each point consists of multiple attribute features.

In ordinary linear regression, parameters are estimated by minimizing the least‑squares loss function, i.e., finding parameters that minimize the sum of squared errors between predicted and actual outputs.

SVR follows the same principle of minimizing a loss function but introduces an ε‑insensitive loss to mitigate over‑fitting. Errors smaller than a predefined ε do not contribute to the loss, effectively ignoring those observations in the objective function.

Mathematically, the SVR problem can be expressed as minimizing a regularized objective with a penalty coefficient \(C\) and the ε‑insensitive loss function:

Introducing slack variables \(\xi_i, \xi_i^*\) leads to the following optimization problem:

By applying Lagrange multipliers, the Lagrangian function is formed, and the corresponding dual problem can be derived.

The solution yields a set of support vectors and associated coefficients that define the regression hyperplane.

References

司守奎，孙玺菁. Python数学实验与建模.