A Comprehensive Guide to Ensemble Learning: Bagging, Boosting, and Stacking

Ensemble Learning Overview

Ensemble learning combines multiple models (weak learners) to build a stronger model with improved accuracy and robustness.

Bias‑Variance Trade‑off

A model must have sufficient capacity to capture data complexity (low bias) while remaining insensitive to training‑set fluctuations (low variance). Ensemble methods aim to reduce either bias or variance, depending on the characteristics of the weak learners.

Bootstrap Sampling

Bootstrap draws B observations with replacement from an original dataset of size N, creating a bootstrap sample that approximates an i.i.d. draw from the true distribution when N is large and B ≪ N. Bootstrap samples are used to estimate variance or confidence intervals of statistical estimators.

Bagging (Bootstrap Aggregating)

Bagging trains L weak learners independently on L bootstrap samples. Because each learner is fitted on an approximately independent dataset, averaging their predictions reduces variance without changing the expected value.

For regression, the ensemble output is the arithmetic mean of the L predictions:

\hat{y}_{ensemble}=\frac{1}{L}\sum_{l=1}^{L}\hat{y}_l

For classification, hard voting selects the majority class, while soft voting averages class probabilities and selects the class with highest average probability. Bagging is naturally parallelizable.

Random Forest

Random forest applies bagging to decision trees and adds random feature sub‑sampling at each split. For each tree, a bootstrap sample is drawn and, at each node, only a random subset of features is considered. Feature sub‑sampling decorrelates trees, improves robustness to missing features, and further reduces variance.

Boosting

Boosting builds an ensemble sequentially. Each new weak learner is trained to focus on observations that previous learners mis‑predicted, thereby reducing bias.

AdaBoost

For binary classification with N observations, AdaBoost maintains a weight w_i for each observation. Initially w_i=1/N. Each iteration performs:

Fit a weak learner that minimizes weighted error.

Compute a coefficient (α) reflecting the learner’s performance.

Add the learner to the ensemble with weight α.

Update observation weights: increase weights of mis‑classified points, decrease weights of correctly classified points.

The final model is a weighted sum of the L weak learners:

Gradient Boosting

Gradient boosting formulates ensemble construction as gradient descent on a differentiable loss function L(y, F(x)). At iteration l:

Compute pseudo‑residuals r_i = -∂L(y_i, F_{l-1}(x_i))/∂F_{l-1}(x_i) for all observations.

Fit a weak learner h_l(x) to the residuals.

Perform a line search to find the optimal step size c_l that minimizes Σ_i L(y_i, F_{l-1}(x_i)+c_l h_l(x_i)).

Update the ensemble: F_l(x)=F_{l-1}(x)+c_l h_l(x).

The process repeats for L iterations, producing a model that is a weighted sum of weak learners and works with arbitrary differentiable loss functions.

Stacking

Stacking combines heterogeneous weak learners (different algorithms) and learns a meta‑model to fuse their predictions.

Split the training data into two disjoint folds.

Fit each of the L chosen weak learners on the first fold.

Generate predictions for the second fold with each weak learner.

Train a meta‑model on these L predictions to produce the final output.

Because the split reduces the amount of data available for training base learners, k‑fold cross‑training can be used: each observation is predicted by models trained on the k‑1 folds that exclude it, yielding meta‑features for the entire dataset.

Multi‑layer Stacking

A multi‑layer stack adds additional levels of meta‑models. In a three‑level stack:

Level 1 fits L base learners.

Level 2 fits M meta‑learners on the Level 1 predictions.

Level 3 fits a final meta‑learner on the M predictions from Level 2.

Key Takeaways

Bagging reduces variance by averaging independent models trained on bootstrap samples; it is suited to low‑bias, high‑variance base learners.

Random forest adds random feature sub‑sampling to bagging, further decorrelating trees and handling missing features.

Boosting reduces bias by sequentially fitting learners that focus on previously mis‑predicted observations; AdaBoost updates observation weights, while gradient boosting fits learners to pseudo‑residuals via gradient descent.

Stacking combines heterogeneous learners using a meta‑model; k‑fold cross‑training mitigates data‑splitting loss, and multi‑layer stacking can deepen the ensemble.