Understanding Bayes’ Theorem: From Basics to Real-World Applications

Bayes’ Theorem

We first look at the magical Bayes’ theorem.

It may seem ordinary, but this is all you need to master Bayes statistics. Understanding how the theorem is derived helps comprehension.

Based on the multiplication rule of probability, we have the following expression, which can be rearranged to obtain Bayes’ theorem:

Bayes’ theorem shows that the probability of a hypothesis given data is proportional to the prior probability of the hypothesis multiplied by the likelihood of the data under that hypothesis.

The theorem highlights that the probability of a hypothesis and the probability of data are not necessarily equal . For example, the probability of being a human given two legs differs from the probability of having two legs given being a human.

In the formula, if we interpret H as a hypothesis and D as data, Bayes’ theorem tells us how to compute the probability of the hypothesis under the given data. Incorporating hypotheses into the theorem requires probability distributions, i.e., treating the hypothesis as a model with a parameter distribution.

Bayes’ theorem is fundamental; we will use it repeatedly. Its components are:

Prior: the distribution reflecting our knowledge of parameters before observing data.

Likelihood: the probability of observing the data given specific parameters.

Posterior: the updated distribution after observing data, proportional to prior times likelihood.

Evidence (marginal likelihood): a normalizing factor representing the average probability of the observed data over all possible parameter values.

The prior can be a uniform distribution when we have no knowledge. Some consider Bayesian statistics subjective, but the prior is merely a modeling assumption, similar in subjectivity to the likelihood.

The likelihood reflects how plausible the observed data are under given parameters.

The posterior represents our complete knowledge after combining data and model; it is a distribution, not a single value. A joke illustrates this: Bayesians may expect a horse, glimpse a donkey, and confidently claim they saw a mule. If both prior and likelihood are vague, the posterior becomes a fuzzy “mule”.

In practice, the posterior from one analysis can serve as the prior for a subsequent analysis, making Bayesian methods well‑suited for sequential data processing such as real‑time weather or satellite monitoring.

Evidence, also called marginal likelihood, is the average probability of the observed data across all parameter values. It often acts as a normalizing constant, allowing us to focus on relative parameter values.

Reference:

Osvaldo Martin, Python Bayesian Analysis