Why Random Forest Beats Linear Regression: Robust Fitting and Clear Feature Importance

Decision‑tree regression works by recursive if‑then splits, fitting piecewise‑constant regions. A single tree is highly sensitive to data variations, making it prone to over‑fitting.

One tree is biased; a forest is more reliable.

Random Forest builds many trees on bootstrap samples and on random subsets of features, then averages their predictions. This bagging reduces variance; the variance of the ensemble is approximately σ²\*((1‑ρ)/N + ρ), where ρ is the correlation between trees and N the number of trees.

Out‑of‑bag (OOB) samples—about one‑third of the data not selected for a given tree—provide an internal estimate of generalization error without a separate validation set.

Complete Example

We generate a synthetic dataset that combines non‑linear functions, feature interactions, heteroscedastic noise, and a small fraction of outliers.

import warnings
warnings.filterwarnings("ignore")
import numpy as np, pandas as pd, matplotlib.pyplot as plt, seaborn as sns
from sklearn.ensemble import RandomForestRegressor
from sklearn.model_selection import train_test_split, RandomizedSearchCV, KFold, learning_curve
from sklearn.metrics import mean_squared_error, mean_absolute_error, r2_score
from sklearn.inspection import PartialDependenceDisplay, permutation_importance

np.random.seed(42)
# 1. Generate data
n = 4000
x0 = np.random.uniform(-3, 3, n)
x1 = np.random.uniform(-2, 2, n)
x2 = np.random.normal(0, 1, n)
x3 = np.random.uniform(-1, 1, n)
x4 = np.random.uniform(0, 2, n)
x5 = np.random.normal(0, 1, n)
x6 = np.random.normal(0, 1, n)
x7 = np.random.uniform(-3, 3, n)
x8 = np.random.uniform(-2, 2, n)

f = (8*np.sin(x0) + 0.8*(x1**2) - 6*np.exp(-(x2**2)) +
     5*(x3>0).astype(float)*x4 + 3*x5*x6 + 4*np.abs(x7) +
     2*np.sin(np.pi*x1*x3) + 0.5*x8**3/6.0)

sigma = 1.0 + 0.7*np.abs(x0) + 0.5*(x3>0).astype(float)*x4
noise = np.random.normal(0, sigma, n)
y = f + noise
# inject outliers
outlier_idx = np.random.choice(np.arange(n), size=int(0.01*n), replace=False)
y[outlier_idx] += np.random.choice([-1, 1], size=len(outlier_idx)) * np.random.uniform(10, 18, size=len(outlier_idx))

X = pd.DataFrame({"x0":x0,"x1":x1,"x2":x2,"x3":x3,"x4":x4,"x5":x5,"x6":x6,"x7":x7,"x8":x8})
y = pd.Series(y, name="y")

Steps followed:

Quick EDA with a pair‑plot of four representative features.

Split data into training (75%) and test (25%) sets.

Fit a baseline RandomForestRegressor (300 trees, max_features=0.7, oob_score=True).

Evaluate on the test set: MSE, MAE, R², and OOB R².

Run a randomized hyper‑parameter search (n_estimators, max_depth, min_samples_leaf, etc.).

Re‑evaluate the best model and compare metrics.

Compute feature importance using impurity‑based (Gini) and permutation methods, visualising both.

Draw partial dependence plots (single‑variable and 2‑D interaction) to interpret marginal effects.

Plot learning curves (training vs. validation R²) to detect over‑/under‑fitting.

Show OOB error versus number of trees to decide when adding more trees stops yielding gains.

Figures (included via <img>) illustrate pairwise feature relationships, prediction‑vs‑true scatter with error‑coloured points, residual hexbin density, importance comparison, PDP curves, learning curves, and OOB error trends.

When to Use Random Forest Regression

Data exhibit strong non‑linearity and complex interactions.

Many noisy or correlated features, possibly with outliers.

A robust, out‑of‑the‑box baseline with some interpretability (feature importance, PDP) is needed.

Training cost is acceptable for moderate‑size datasets.

Limitations:

Poor extrapolation beyond the range of training data.

Higher computational cost on very large datasets.

Not optimal for extremely high‑dimensional sparse data (e.g., text) compared with linear models or gradient‑boosted trees.

Overall, Random Forest regression provides an industrial‑grade, reliable baseline that balances predictive power and interpretability for most practical regression problems.