In Chapters [ch:thinking]–[ch:theorem] the hypotheses were discrete (sick or healthy). The real power of Bayesian inference appears when the unknown is a continuous parameter — a probability, a rate, a mean — so the prior and posterior become distributions over a parameter. This chapter is the conceptual engine of the book: how the three ingredients combine for continuous parameters, why the posterior is a compromise, and how it sharpens with data.

Inference about a parameter

Let \(\theta\) be an unknown parameter (say, the probability a coin lands heads). Bayesian inference produces the full posterior density \[p(\theta\mid D)=\frac{p(D\mid\theta)\,p(\theta)}{p(D)},\qquad p(D)=\int p(D\mid\theta)\,p(\theta)\,d\theta,\] where \(p(\theta)\) is the prior density, \(p(D\mid\theta)\) the likelihood, and the integral \(p(D)\) the evidence. The answer is the whole curve \(p(\theta\mid D)\), not a single number.

The likelihood function

Fix the data and view \(p(D\mid\theta)\) as a function of \(\theta\): this is the likelihood function. It scores every parameter value by how well it predicts what we saw. For \(n\) independent observations it is a product, \[p(D\mid\theta)=\prod_{i=1}^n p(x_i\mid\theta),\] so we usually work with the log-likelihood (sums beat products numerically). The likelihood is the data’s entire contribution to inference — the bridge between model and evidence.

For a coin with \(h\) heads in \(n\) flips, the likelihood is \(\theta^{h}(1-\theta)^{n-h}\) — peaked at \(\theta=h/n\), the value that best explains the data. The maximum likelihood estimate (MLE) sits at that peak. A Bayesian does not just take the peak; they multiply the whole curve by the prior and keep the result as a distribution, retaining the uncertainty the peak alone discards.

Prior \(\times\) likelihood \(=\) (unnormalized) posterior

The posterior sits between the prior and the likelihood — a weighted compromise. Its location is pulled toward whichever is more confident (more sharply peaked), and its spread is narrower than either, because combining two sources of information always reduces uncertainty. This single picture explains the entire behavior of Bayesian updating.

How the posterior evolves with data

Two limits make the compromise concrete:

  • Little data: the likelihood is broad, so the prior matters a lot — the posterior stays close to the prior.

  • Much data: the likelihood becomes sharp and dominates; the posterior concentrates near the MLE and the prior’s influence fades (the Bernstein–von Mises phenomenon).

So priors are training wheels: indispensable when data is scarce, almost irrelevant once data is abundant. This is why reasonable people with different priors converge once they see enough evidence.

The predictive distribution

To predict a new observation \(\tilde x\), average the likelihood over the posterior — the posterior predictive distribution: \[p(\tilde x\mid D)=\int p(\tilde x\mid\theta)\,p(\theta\mid D)\,d\theta.\]

This is a defining Bayesian move: instead of plugging in one “best” \(\theta\), we marginalize (average) over all values weighted by their posterior probability. Predictions therefore inherit parameter uncertainty — a Bayesian forecast is automatically wider when the parameter is poorly known, which is exactly the calibrated behavior we want.

Do not collapse the posterior to its peak too early. Reporting only the MAP or posterior mean throws away the uncertainty that is the whole point; and plugging a point estimate into predictions (rather than marginalizing) systematically understates predictive uncertainty. Keep the distribution as long as you can, and integrate, not maximize, when you predict.

The likelihood is the same object that supervised learning minimizes: negative log-likelihood is the loss function (MSE for Gaussian, cross-entropy for categorical). Maximum likelihood is the non-Bayesian special case of taking the likelihood peak; adding a prior and taking the peak gives MAP \(=\) regularized training (Chapter [ch:bayesmlai]). And the posterior-predictive “marginalize, don’t maximize” principle is precisely what Bayesian neural networks, deep ensembles, and MC-dropout approximate to get trustworthy uncertainty — averaging predictions over many plausible parameter settings rather than betting on one.

import numpy as np
from scipy.stats import beta
h, n = 7, 10                       # 7 heads in 10 flips
grid = np.linspace(0, 1, 1001)
prior = beta(4, 4).pdf(grid)        # belief: probably near fair
like  = grid**h * (1-grid)**(n-h)   # binomial likelihood (unnormalized)
post  = prior * like
post /= np.trapz(post, grid)        # normalize via the evidence integral
print("posterior mean:", round(np.trapz(grid*post, grid), 3))   # compromise value
# Posterior-predictive prob the next flip is heads = posterior mean of theta

Chapter summary

  • For a continuous parameter, prior and posterior are densities; \(p(\theta\mid D)\propto p(D\mid\theta)p(\theta)\) with evidence \(p(D)=\int p(D\mid\theta)p(\theta)d\theta\).

  • The likelihood \(p(D\mid\theta)\) scores parameters by fit; its peak is the MLE, and it is a product (use log-likelihood).

  • The posterior is a compromise between prior and likelihood, pulled toward the sharper one and narrower than both.

  • With more data the likelihood dominates and the prior’s influence fades; priors matter most when data is scarce.

  • Predict by the posterior predictive — marginalize over \(\theta\), don’t plug in one value — which is what Bayesian deep learning approximates.