What Classic Diagrams Reveal About Test Error, Overfitting, and Model Selection

Key Machine Learning Diagrams

When explaining basic machine‑learning concepts, I often return to a handful of illustrative diagrams. Below is a list of the most insightful ones.

Test and training error

Why a low training error is not always desirable: the figure shows test and training error curves as model complexity varies.

Under and overfitting

Examples of under‑fitting and over‑fitting: polynomial curves of varying degree (M) are shown in red, with the green curve fitting the data.

Occam’s razor

The diagram explains how Bayesian inference embodies Occam’s razor: a simple model (H1) has higher evidence for a given dataset than a more complex model (H2) when both have equal priors.

Feature combinations

Why individually irrelevant but jointly correlated features matter, and why linear methods may fail, as illustrated in Isabelle Guyon’s feature‑extraction slides.

Irrelevant features

Irrelevant features can degrade K‑Nearest Neighbors, clustering, and other similarity‑based methods; the right‑hand plot adds an unrelated axis that disrupts grouping.

Basis functions

Non‑linear basis functions transform a low‑dimensional non‑linear classification problem into a high‑dimensional linear one, as shown in Andrew Moore’s SVM tutorial (e.g., mapping x to (x, x²)).

Discriminative vs. Generative

Discriminative learning is often simpler: the left plot shows class‑conditional density p(x|C₁) (blue curve) which does not affect posterior probabilities, while the right plot shows the decision boundary (green line) that minimizes error.

Loss functions

Learning algorithms can be viewed as optimizing different loss functions: the hinge loss for SVM (blue), a scaled loss for logistic regression (red), misclassification loss (black), and mean‑squared error (green).

Geometry of least squares

The figure shows the N‑dimensional geometry of least‑squares regression with two predictors: the response vector y is orthogonally projected onto the plane spanned by input vectors x₁ and x₂.

Sparsity

Lasso (L₁ regularization or Laplace prior) yields sparse solutions with many zero coefficients. The left plot shows the Lasso estimate, the right plot shows ridge regression; the red ellipse represents the least‑squares error contours, while the blue region denotes the constraint |β₁|+|β₂| ≤ t (Lasso) or β₁²+β₂² ≤ t² (ridge).