Maximum Likelihood Estimation Explained Simply: Core Principles, Code Walkthroughs, and Limitations

Core Idea of Maximum Likelihood Estimation

Maximum likelihood estimation (MLE) chooses the parameter values that make the observed data appear "least surprising" under a chosen statistical model. The article starts with a simple coin‑flip experiment (seven heads, three tails) to illustrate how MLE selects the bias p that best explains the data.

Probability vs. Likelihood

The author uses a "director vs. detective" metaphor: probability predicts data given known rules (the director), while likelihood infers the most plausible hidden rule after the data are observed (the detective). For the coin, the likelihood of p = 0.5 is 0.5¹⁰ ≈ 0.000977, whereas p = 0.7 yields 0.7⁷·0.3³ ≈ 0.00222, making the latter explanation less surprising.

Log‑Likelihood and Numerical Stability

Multiplying many small probabilities quickly underflows to zero. By taking the natural logarithm, products become sums, preserving the ordering of likelihood values while avoiding underflow. The log‑likelihood curve retains the same maximum at p = 0.7.

Python Demonstration – Discrete Case

import numpy as np
flips = np.array([1,1,0,1,1,0,1,1,1,0])  # 1=heads, 0=tails
heads, total = flips.sum(), flips.size

def log_likelihood(p):
    return heads * np.log(p) + (total - heads) * np.log(1 - p)

candidate_p = np.linspace(0.001, 0.999, 1000)
scores = log_likelihood(candidate_p)
best_p = candidate_p[np.argmax(scores)]
print(round(best_p, 3))  # 0.7

A brute‑force grid search finds the peak at p ≈ 0.7, matching intuition.

Optimized Search

from scipy.optimize import minimize_scalar

def neg_log_likelihood(p):
    return -(heads * np.log(p) + (total - heads) * np.log(1 - p))

result = minimize_scalar(neg_log_likelihood, bounds=(0.001, 0.999), method="bounded")
print(round(result.x, 3))  # 0.7

The optimizer climbs the same hill more efficiently.

Extension to Continuous Distributions

For ten height measurements, assuming a normal distribution, MLE yields the sample mean and standard deviation as the most likely parameters:

import numpy as np
data = np.array([171,165,180,158,169,174,162,177,168,173])
mu_hat = data.mean()
sigma_hat = data.std()
print(round(mu_hat,2), round(sigma_hat,2))

Thus, ordinary averaging is a special case of MLE.

Historical Note

Ronald Fisher introduced the likelihood concept in the early 1920s, arguing that inference should let the data speak rather than imposing prior guesses. His work laid the foundation for modern statistical inference.

Limitations of MLE

MLE can be misleading with tiny samples (e.g., two heads → p = 1) and tends to over‑fit flexible models. For normal distributions, the variance estimate is slightly biased downward, a bias that diminishes with more data.

Connection to Machine Learning

Many machine learning loss functions are negative log‑likelihoods. Logistic regression, spam filters, and language models all maximize likelihood (or equivalently minimize negative log‑likelihood) to fit parameters, turning the statistical “hill‑climbing” into a loss‑minimization problem.

Practical Takeaway

Understanding MLE equips you to recognize its role behind average calculations, logistic regression, and deep‑learning training, while reminding you to guard against small‑sample pitfalls and over‑fitting.