Frequentist statistics treats parameters as fixed unknowns and data as random. Bayesian inference flips the emphasis: it treats parameters as random variables with probability distributions that encode our beliefs, and updates those beliefs with data via Bayes’ theorem. The result is a complete distribution over what we don’t know — not a single point — which is precisely the language of uncertainty that modern, safety-conscious AI needs.
The Bayesian recipe
Bayesian inference computes the posterior distribution of parameters given data: \[\underbrace{p(\theta\given\Ldata)}_{\text{posterior}}=\frac{\overbrace{p(\Ldata\given\theta)}^{\text{likelihood}}\;\overbrace{p(\theta)}^{\text{prior}}}{\underbrace{p(\Ldata)}_{\text{evidence}}},\qquad p(\Ldata)=\int p(\Ldata\given\theta)p(\theta)\,\dd\theta.\] In words: posterior \(\propto\) likelihood \(\times\) prior.
The shift from MLE to Bayes is the shift from “the single best \(\theta\)” to “the full distribution of plausible \(\theta\).” The prior \(p(\theta)\) states what you believe before seeing data; the likelihood updates it; the posterior is your refreshed belief. With more data the likelihood dominates and the prior washes out — but with little data the prior is what keeps you sane (and what prevents overfitting). Everything you might want — a point estimate, an interval, a prediction — is read off the posterior.
Conjugate priors: Bayes in closed form
For certain prior–likelihood pairs the posterior stays in the same family as the prior — a conjugate prior — making the update pure algebra.
To estimate a coin’s bias \(p\), use a \(\Betad(\alpha,\beta)\) prior. After observing \(k\) heads in \(n\) flips, the posterior is simply \[p\given\Ldata \sim \Betad(\alpha+k,\ \beta+n-k).\] The prior parameters act like “pseudo-counts” of heads and tails seen in advance. Start with \(\Betad(1,1)\) (uniform) and the posterior mean is \(\frac{k+1}{n+2}\) — Laplace smoothing, the same \(+1\) that keeps naive Bayes and language models from assigning zero probability to unseen events.
The Gaussian is conjugate to itself (Gaussian prior + Gaussian likelihood \(\Rightarrow\) Gaussian posterior), which is why linear-Gaussian models and Kalman filters update in closed form.
Point estimates from the posterior
The posterior is a whole distribution; to summarize it:
The posterior mean \(\E[\theta\given\Ldata]\) minimizes expected squared error.
The MAP estimate \(\hat\theta_{\text{MAP}}=\argmax_\theta p(\theta\given\Ldata)=\argmax_\theta\big[\log p(\Ldata\given\theta)+\log p(\theta)\big]\) is the posterior’s peak.
MAP \(=\) MLE \(+\) prior penalty, and that penalty is regularization. With a Gaussian prior \(\theta\sim\Normal(0,\tau^2)\), \(\log p(\theta)\) contributes a term \(\propto-\|\theta\|^2\) — exactly L2 regularization / weight decay (ridge regression). With a Laplace prior, it contributes \(\propto-\|\theta\|_1\) — exactly L1 / lasso, which induces sparsity. So every weight penalty in deep learning is, secretly, a prior belief that parameters should be small. This is the single most important bridge between Bayesian statistics and practical ML.
Credible intervals and prediction
A credible interval is an interval containing, say, \(95\%\) of the posterior probability — and it means exactly what people wish a confidence interval meant: “given the data and prior, there is a \(95\%\) probability the parameter lies here.” To predict new data we use the posterior predictive, averaging over all parameter values weighted by the posterior: \[p(x_{\text{new}}\given\Ldata)=\int p(x_{\text{new}}\given\theta)\,p(\theta\given\Ldata)\,\dd\theta.\] This marginalization over uncertainty is what gives Bayesian predictions their honest, often better-calibrated confidence.
Bayesian vs. frequentist
The two schools answer different questions. Frequentist: “what procedure has good long-run error rates?” Bayesian: “what should I believe given this data and my prior?” Frequentist methods need no prior and dominate hypothesis testing; Bayesian methods handle small data, prior knowledge, and uncertainty propagation gracefully but require choosing a prior and (usually) heavy computation for the evidence integral. Modern ML is pragmatically both: point estimates by penalized MLE (a MAP in disguise), with Bayesian ideas supplying the regularizers, the uncertainty estimates, and the generative models.
Bayesian thinking is woven through ML even when no one says “Bayes.” Weight decay is a Gaussian prior; L1 sparsity is a Laplace prior; dropout approximates Bayesian model averaging; label smoothing is a prior on the label distribution. Bayesian neural networks, MC dropout, and deep ensembles approximate the posterior predictive to produce calibrated uncertainty — crucial for medicine, autonomy, and active learning. Variational inference (Chapter [ch:information]) makes the intractable evidence integral tractable and trains VAEs. Gaussian processes are fully Bayesian nonparametric models. When a system says “I am \(80\%\) sure,” and means it, there is a posterior underneath.
import numpy as np
from scipy import stats
# Beta-Bernoulli conjugate update for a coin's bias
alpha, beta = 1, 1 # uniform prior
heads, tails = 7, 3
post = stats.beta(alpha+heads, beta+tails)
print(round(post.mean(), 3)) # (7+1)/(10+2) = 0.667 (Laplace smoothing)
print(post.interval(0.95)) # 95% CREDIBLE interval for p
# MAP with a Gaussian prior == ridge: argmax [loglik - lambda*||theta||^2]
# (the prior's log-term is exactly the L2 penalty)Chapter summary
Bayesian inference computes a posterior \(\propto\) likelihood \(\times\) prior — a full distribution over parameters.
Conjugate priors (Beta–Bernoulli, Gaussian–Gaussian) give closed-form updates; pseudo-counts yield Laplace smoothing.
MAP \(=\) MLE \(+\) prior: Gaussian prior \(\to\) L2/weight decay, Laplace prior \(\to\) L1/lasso. Regularization is a prior.
Credible intervals mean what CIs are mistaken for; the posterior predictive marginalizes over uncertainty.
Bayesian ideas power weight decay, dropout, ensembles, BNNs, variational inference, and Gaussian processes — the machinery of calibrated uncertainty.