The Timeless Foundations of Machine Learning: 6 Core Algorithms Explained

Recently, Andrew Ng updated a post in his AI newsletter "The Batch," summarizing the historical origins of several fundamental machine‑learning algorithms.

He recalls a past project where he chose between neural networks and decision‑tree learning, opting for the former due to computational budget, a decision later corrected by his team.

Ng emphasizes the importance of continually revisiting basic knowledge, noting that while many algorithms evolve, a few core ideas endure:

Algorithms: linear and logistic regression, decision trees, etc.

Concepts: regularization, loss‑function optimization, bias/variance trade‑offs.

He identifies six algorithms that remain central to many models, from house‑price predictors to text‑image generators like DALL·E:

Linear Regression

First formulated by French mathematician Adrien‑Marie Legendre in 1805 for fitting a line to data points, later claimed by Carl Friedrich Gauss. It models relationships such as vehicle fuel consumption versus weight using a line y = w·x + b, optimized via ordinary least squares and extended with regularization (ridge, lasso, elastic net).

Logistic Regression

Originally used to model binary outcomes such as survival after poison exposure, its roots trace back to Verhulst’s 1830s population curves and later formalized by E. B. Wilson and Jane Worcester. It fits a sigmoid function to estimate probabilities, with extensions for multi‑class and ordered outcomes, and can incorporate regularization similar to linear regression.

Gradient Descent

Analogous to walking downhill to find the lowest point, the method iteratively adjusts model parameters to minimize a loss function, guided by the gradient and a learning‑rate step size. Variants address issues like local minima, saddle points, and non‑convex landscapes.

Neural Networks

Inspired by biological neurons, the first mathematical model appeared in 1943 (McCulloch & Pitts) and the perceptron in 1958 (Rosenblatt). Stacking layers (Ivakhnenko, Lapa) and training via back‑propagation (LeCun, Rumelhart, et al.) enabled deep learning, now powered by GPUs and large datasets.

Decision Trees

Early logical classifications date back to Aristotle’s categories; modern computational trees emerged in the 1960s (Sonquist & Morgan) and were popularized by algorithms like ID3, C4.5, and random forests (Breiman). Trees split data based on feature purity, but can overfit without pruning.

K‑means Clustering

Proposed by Stuart Lloyd in 1957 (originally for signal quantization) and later popularized as the Lloyd‑Forgy algorithm, it partitions data into k groups by iteratively updating centroids to the mean of assigned points. Variants such as k‑medoids and fuzzy C‑means address centroid selection and soft assignments.

These six algorithms—linear regression, logistic regression, gradient descent, neural networks, decision trees, and k‑means clustering—form the enduring backbone of machine learning, influencing a wide range of modern AI systems.