Almost all useful probability is conditional: we want to know the chance of something given what we already know. How likely is disease given a positive test? Spam given the word “free”? The next word given the sentence so far? Conditioning is how probability incorporates evidence, and Bayes’ theorem is the engine that reverses it — turning “probability of evidence given cause” into “probability of cause given evidence.” This is the single most important chapter for understanding machine learning as inference.

Conditional probability

The conditional probability of \(A\) given \(B\) (with \(P(B)>0\)) is \[P(A\given B)=\frac{P(A\cap B)}{P(B)}.\]

Conditioning shrinks the sample space. Once you know \(B\) happened, \(B\) becomes your new universe of possibilities; you ask what fraction of that universe also has \(A\). Dividing by \(P(B)\) renormalizes so the conditional probabilities still sum to one within \(B\). Learning “\(B\) occurred” is exactly how a model updates when it sees data.

The multiplication and chain rules

Rearranging the definition gives the multiplication rule \(P(A\cap B)=P(A\given B)P(B)\), which extends to the chain rule: \[P(A_1\cap\cdots\cap A_n)=P(A_1)\,P(A_2\given A_1)\,P(A_3\given A_1,A_2)\cdots P(A_n\given A_1,\dots,A_{n-1}).\] This factorization — breaking a complex joint probability into a product of simpler conditionals — is the backbone of probabilistic models, from naive Bayes to autoregressive language models that generate text one token at a time.

Independence

Events \(A\) and \(B\) are independent (\(A\indep B\)) if knowing one tells you nothing about the other: \[P(A\cap B)=P(A)\,P(B),\quad\text{equivalently}\quad P(A\given B)=P(A).\]

Independence is a strong, often-violated assumption — and “independent” is not the same as “mutually exclusive.” Disjoint events with positive probability are in fact maximally dependent (if one happens, the other cannot). Also beware conditional independence: \(A\) and \(B\) can be dependent overall yet independent given \(C\) (or vice versa). Naive Bayes assumes features are conditionally independent given the class; it works surprisingly well even though the assumption is usually false.

The law of total probability

To find the probability of \(A\), split the world into exhaustive, disjoint cases \(B_1,\dots,B_n\) and average: \[P(A)=\sum_{i} P(A\given B_i)\,P(B_i).\] This “condition on every possibility and recombine” move is ubiquitous — it is the discrete cousin of marginalizing (integrating out) a variable, which is how we compute the evidence term below.

Bayes’ theorem

\[\underbrace{P(H\given E)}_{\text{posterior}}=\frac{\overbrace{P(E\given H)}^{\text{likelihood}}\;\overbrace{P(H)}^{\text{prior}}}{\underbrace{P(E)}_{\text{evidence}}},\qquad P(E)=\sum_i P(E\given H_i)P(H_i).\]

Bayes’ theorem is how rational belief updates with evidence. Start with a prior \(P(H)\) (your belief in hypothesis \(H\) before data); multiply by the likelihood \(P(E\given H)\) (how well \(H\) predicts the evidence \(E\)); normalize by the evidence \(P(E)\); out comes the posterior \(P(H\given E)\) (your updated belief). Posterior \(\propto\) likelihood \(\times\) prior. This one line is the foundation of Bayesian inference, of spam filters, of medical diagnosis, and of the entire probabilistic view of learning.

A disease affects \(1\%\) of people. A test is \(99\%\) accurate (both sensitivity and specificity). You test positive — what is the chance you are sick? Intuition screams “\(99\%\),” but Bayes says otherwise. With prior \(P(H)=0.01\): \[P(H\given +)=\frac{0.99\times0.01}{0.99\times0.01+0.01\times0.99}=\frac{0.0099}{0.0198}=0.5.\] Only \(50\%\). Because the disease is rare, false positives from the healthy \(99\%\) are as numerous as true positives. Ignoring the base rate (the prior) is one of the most consequential reasoning errors — in medicine, in law, and in ML systems that flag rare events.

The lesson of the base-rate fallacy: the prior matters. Evidence shifts belief, but it shifts it from where you started. A very rare hypothesis needs very strong evidence to become probable. This is exactly why class imbalance wrecks naive classifiers, why anomaly detectors drown in false alarms, and why calibrated priors are essential whenever the thing you are detecting is rare.

Bayes’ theorem is the probabilistic template for learning. Replace \(H\) with model parameters \(\vtheta\) and \(E\) with data \(\Ldata\): \[P(\vtheta\given\Ldata)\;\propto\;P(\Ldata\given\vtheta)\,P(\vtheta).\] Maximizing the likelihood \(P(\Ldata\given\vtheta)\) alone is maximum likelihood estimation (Chapter [ch:estimation]) — the origin of cross-entropy and squared-error losses. Maximizing likelihood times a prior is MAP estimation — the origin of weight decay and other regularizers (Chapter [ch:bayesian]). The chain rule \(P(x_1,\dots,x_n)=\prod_t P(x_t\given x_{<t})\) is precisely how autoregressive language models (GPT-style) factor the probability of a sentence. Conditioning, independence, and Bayes are not background math — they are the specification of what these models compute.

# Base-rate fallacy: P(sick | positive) with a rare disease
prior_sick = 0.01
sens = 0.99          # P(positive | sick)
spec = 0.99          # P(negative | healthy)  => P(positive | healthy) = 0.01
p_pos = sens*prior_sick + (1-spec)*(1-prior_sick)   # law of total probability
posterior = sens*prior_sick / p_pos
print(round(posterior, 3))     # 0.5  -- only 50%, not 99%!

# Bayesian updating: a second independent positive test
prior2 = posterior
p_pos2 = sens*prior2 + (1-spec)*(1-prior2)
print(round(sens*prior2 / p_pos2, 3))   # 0.99 -- evidence accumulates

Chapter summary

  • Conditional probability \(P(A\given B)=P(A\cap B)/P(B)\) shrinks the sample space to \(B\).

  • The chain rule factors joint probabilities into products of conditionals — the basis of autoregressive models.

  • Independence means \(P(A\cap B)=P(A)P(B)\); it differs from mutual exclusivity, and conditional independence is subtle.

  • The law of total probability averages over exhaustive cases; it computes the evidence term.

  • Bayes’ theorem (posterior \(\propto\) likelihood \(\times\) prior) updates belief with evidence; ignoring the prior causes the base-rate fallacy. With \(\vtheta\) and \(\Ldata\) it becomes the template for MLE and MAP learning.