Mastering Model Evaluation: Key Metrics, Validation Techniques, and Diagnostics

Metrics

Given a set of data points, each with n features and associated outputs, we need to evaluate the performance of a classifier.

Classification

In binary classification, several important metrics are used to assess model performance.

Confusion Matrix

The confusion matrix summarizes the predictions of a classification model in an NxN table.

Main Metrics

These metrics are commonly used to evaluate classification models.

ROC

The Receiver Operating Characteristic curve shows the trade‑off between true positive rate (TPR) and false positive rate (FPR) as the decision threshold varies.

AUC

The Area Under the ROC Curve (AUC or AUROC) quantifies the overall ability of the model to discriminate between classes.

Regression

Basic Metrics

For regression models, the following metrics are typically used to assess performance.

Coefficient of Determination

The coefficient of determination, expressed as R² or r², measures how well the observed outcomes fit the model.

Main Metrics

These metrics evaluate regression model performance, taking into account the number of variables n.

When L represents likelihood, it provides the variance estimate for each response.

Model Selection

Vocabulary

When selecting a model, we divide the dataset into three parts:

Training set – used for model training, typically 80% of the data. Validation set – used for model performance evaluation, typically 20% of the data, also called hold‑out or development set. Test set – used for final model prediction on unseen data.

After a model is chosen, it is retrained on the entire dataset and evaluated on the unseen test set.

Cross‑validation

Cross‑validation (CV) is a technique that reduces dependence on the initial training set. Common types include:

k‑fold: train on k‑1 folds and validate on the remaining fold; typical k values are 5 or 10.

Leave‑p‑out: train on n‑p samples and validate on the remaining p samples; when p=1, it is leave‑one‑out.

The most widely used method is k‑fold cross‑validation, where the data are split into k parts, each part serving once as the validation set while the others are used for training. The average error over the k runs is the cross‑validation error.

Regularization

Regularization aims to prevent overfitting and address high variance. Common regularization techniques are illustrated below.

In practice, we train the model on the training set, evaluate it on the validation set, select the best‑performing model, and finally retrain that model on the entire training data.

Diagnostics

Bias

Bias measures the difference between the model’s predictions and the true values for a given data point.

Variance

Variance reflects the variability of the model’s predictions for a given data point across different training runs, indicating model stability.

Bias / Variance Tradeoff

A simpler model tends to have higher bias, while a more complex model tends to have higher variance.