Now we make the update rule exact. Bayes’ theorem is a short consequence of the definition of conditional probability, but its implications are deep enough to have reshaped statistics, science, and AI. This chapter derives it, names its parts, and works the classic medical-test example that exposes the most common reasoning error about probability — the base-rate fallacy.

From conditional probability to Bayes

The conditional probability of \(A\) given \(B\) is \(P(A\mid B)=\dfrac{P(A\cap B)}{P(B)}\). Writing the joint probability two ways, \(P(A\cap B)=P(A\mid B)P(B)=P(B\mid A)P(A)\), and rearranging gives Bayes’ theorem: \[\boxed{\,P(A\mid B)=\frac{P(B\mid A)\,P(A)}{P(B)}\,}.\]

Bayes’ theorem reverses a conditional. It converts \(P(B\mid A)\) — often the thing we can model or measure (how the cause produces the effect) — into \(P(A\mid B)\), the thing we want to know (the cause, given the observed effect). In inference we write it with a hypothesis \(\Hyp\) and data \(D\): \[P(\Hyp\mid D)=\frac{P(D\mid \Hyp)\,P(\Hyp)}{P(D)},\qquad \underbrace{P(\Hyp\mid D)}_{\text{posterior}}=\frac{\overbrace{P(D\mid \Hyp)}^{\text{likelihood}}\ \overbrace{P(\Hyp)}^{\text{prior}}}{\underbrace{P(D)}_{\text{evidence}}}.\]

The four parts

Belief in the hypothesis before seeing data.

How probable the observed data is if the hypothesis holds. (Read as a function of \(\Hyp\) for fixed \(D\); it is not a probability distribution over \(\Hyp\).)

The total probability of the data under all hypotheses, \(P(D)=\sum_i P(D\mid \Hyp_i)P(\Hyp_i)\) (or an integral). It is a normalizing constant ensuring the posterior sums to one.

The updated belief — the goal of the analysis.

Because the evidence \(P(D)\) does not depend on \(\Hyp\), it only rescales; the shape of the posterior comes entirely from likelihood \(\times\) prior. That is why the working form of Bayes is the proportionality \(P(\Hyp\mid D)\propto P(D\mid\Hyp)P(\Hyp)\) — compute that, then normalize at the end. This single simplification saves enormous effort and is the reason conjugate priors (Chapter [ch:conjugate]) are so convenient.

The medical-test example

A disease affects \(1\%\) of a population. A test is \(99\%\) sensitive (\(P(+\mid \text{disease})=0.99\)) and \(95\%\) specific (\(P(-\mid \text{healthy})=0.95\), so the false-positive rate is \(5\%\)). You test positive. What is the probability you have the disease? \[P(\text{dis}\mid +)=\frac{P(+\mid\text{dis})P(\text{dis})}{P(+\mid\text{dis})P(\text{dis})+P(+\mid\text{healthy})P(\text{healthy})} =\frac{0.99\times0.01}{0.99\times0.01+0.05\times0.99}\approx 0.167.\] Despite the “99% accurate” test, there is only a 17% chance you are actually sick.

This is the base-rate fallacy: ignoring the prior \(P(\Hyp)\) and confusing \(P(D\mid\Hyp)\) with \(P(\Hyp\mid D)\). The test’s accuracy (\(P(+\mid\text{dis})\)) is not the chance of disease given a positive (\(P(\text{dis}\mid +)\)). When the base rate is low, even a good test produces mostly false positives, because there are so many more healthy people to generate them. Forgetting the base rate corrupts reasoning in medicine, law (the “prosecutor’s fallacy”), and spam filtering alike — always anchor on the prior.

Sequential updating

Bayes’ theorem chains. Apply it to datum \(D_1\) to get a posterior; treat that posterior as the prior for \(D_2\); and so on. With independent data the result is the same as updating on all data at once: \[P(\Hyp\mid D_1,D_2)\propto P(D_2\mid\Hyp)\,P(D_1\mid\Hyp)\,P(\Hyp).\] Today’s posterior is tomorrow’s prior — which makes Bayesian learning naturally online and incremental.

Bayes’ theorem is the literal engine of many ML systems. The original spam filters scored emails by \(P(\text{spam}\mid\text{words})\) via Bayes’ rule (Chapter [ch:naivebayes]). Bayesian filtering (Kalman filters, particle filters) tracks robots and prices by repeated predict-update steps. In probabilistic ML the same equation links observations to latent variables, and the awkward denominator \(P(D)\) — the evidence — is precisely the intractable quantity that MCMC and variational inference (Chapter [ch:mcmc]) exist to approximate, and that doubles as the score for Bayesian model comparison (Chapter [ch:modelcomp]).

def posterior_disease(prior, sens, spec):
    p_pos = sens*prior + (1-spec)*(1-prior)         # total prob of testing +
    return sens*prior / p_pos
print(round(posterior_disease(0.01, 0.99, 0.95), 3))   # 0.167
# Retest positive again -> yesterday's posterior becomes today's prior
p1 = posterior_disease(0.01, 0.99, 0.95)
print(round(posterior_disease(p1, 0.99, 0.95), 3))     # ~0.799 after 2nd positive

Chapter summary

  • Bayes’ theorem, \(P(\Hyp\mid D)=P(D\mid\Hyp)P(\Hyp)/P(D)\), reverses a conditional: from “effect given cause” to “cause given effect.”

  • Its parts are prior, likelihood, evidence (normalizer), and posterior; in practice use posterior \(\propto\) likelihood \(\times\) prior.

  • The base-rate fallacy: a 99%-accurate test on a 1%-prevalent disease yields only \(\sim\)17% posterior — the prior cannot be ignored.

  • Bayes chains: today’s posterior is tomorrow’s prior, enabling online learning.

  • The evidence \(P(D)\) is the hard-to-compute term that motivates MCMC/VI and scores models.