Computing the posterior means doing the integral \(p(D)=\int p(D\mid\theta)p(\theta)\,d\theta\) — usually hard. Conjugate priors are a beautiful special case where the integral is free: prior and posterior belong to the same family, so updating is just arithmetic on the parameters. The three classic conjugate pairs in this chapter are the workhorses of analytic Bayes and the clearest illustration of “the posterior is prior plus data.”

Conjugacy

A prior is conjugate to a likelihood if the resulting posterior is in the same family as the prior. Then Bayesian updating reduces to mapping the prior’s parameters to the posterior’s parameters — no integration required.

Conjugacy turns the calculus of Bayes’ theorem into bookkeeping. You start with a prior parameterized by a few numbers (“pseudo-counts”), you observe data summarized by a few sufficient statistics, and the posterior parameters are simply prior \(+\) data. The intuition is irresistible: the prior acts like data you have already seen.

Beta–Binomial (a probability)

For a success probability \(\theta\) with a Binomial likelihood, the conjugate prior is the Beta distribution. With prior \(\theta\sim\Betad(\alpha,\beta)\) and data of \(h\) successes in \(n\) trials: \[\theta\mid D\ \sim\ \Betad(\alpha+h,\ \beta+n-h).\]

Read \(\alpha\) as “prior successes” and \(\beta\) as “prior failures” — pseudo-counts encoding belief before data. Updating just adds the observed counts. The posterior mean \(\frac{\alpha+h}{\alpha+\beta+n}\) is a weighted average of the prior mean and the data proportion \(h/n\), with the prior’s weight set by \(\alpha+\beta\) (its “confidence in trials”). A flat \(\Betad(1,1)\) prior gives posterior mean \(\frac{h+1}{n+2}\) — Laplace’s rule of succession, the original smoothing.

Gamma–Poisson (a rate)

For a rate \(\lambda\) of events with a Poisson likelihood, the conjugate prior is the Gamma. With prior \(\lambda\sim\Gammad(\alpha,\beta)\) (shape \(\alpha\), rate \(\beta\)) and observed total count \(\sum x_i\) over \(n\) periods: \[\lambda\mid D\ \sim\ \Gammad\!\Big(\alpha+\textstyle\sum_i x_i,\ \beta+n\Big).\]

Here \(\alpha\) is “prior total events” and \(\beta\) is “prior periods observed.” Updating adds the events seen to \(\alpha\) and the periods elapsed to \(\beta\) — again, prior pseudo-counts plus data. The posterior mean rate \(\frac{\alpha+\sum x_i}{\beta+n}\) blends the prior rate with the empirical rate. Use this for counts: server requests per minute, shark attacks per year, photons per pixel.

Normal–Normal (a mean)

For an unknown mean \(\mu\) with known variance \(\sigma^2\) and a Normal likelihood, the conjugate prior is Normal. With prior \(\mu\sim\Normal(\mu_0,\tau_0^2)\) and \(n\) observations of sample mean \(\bar x\), the posterior is Normal with precision (inverse variance) adding: \[\frac{1}{\tau_n^2}=\frac{1}{\tau_0^2}+\frac{n}{\sigma^2},\qquad \mu_n=\tau_n^2\!\left(\frac{\mu_0}{\tau_0^2}+\frac{n\bar x}{\sigma^2}\right).\]

Precisions add, and the posterior mean is a precision-weighted average of the prior mean and the data mean. Each new observation adds \(1/\sigma^2\) to the precision, so certainty grows linearly with \(n\) and the posterior standard deviation shrinks like \(1/\sqrt n\) — the Bayesian echo of the standard error.

Why it works: the exponential family

Conjugacy is not a coincidence: every likelihood in the exponential family (Bernoulli, Binomial, Poisson, Normal, Gamma, …) has a conjugate prior, because multiplying prior and likelihood just adds their natural parameters. This is the same exponential-family structure that unifies generalized linear models — statistics’ deep patterns rhyme.

Conjugate priors are chosen for mathematical convenience, which can conflict with honest belief. If your real prior knowledge is not Beta/Gamma/Normal-shaped, forcing conjugacy distorts it. Conjugacy also covers only simple models; realistic multi-parameter models rarely have closed-form posteriors — which is exactly why MCMC and variational inference (Chapter [ch:mcmc]) were invented. Treat conjugacy as a valuable special case and a sanity check, not the whole toolbox.

Conjugate updates are the analytic core of many ML methods. Naive Bayes text classifiers use Beta/Dirichlet smoothing of word counts — Laplace smoothing is a Beta prior (Chapter [ch:naivebayes]). Thompson sampling, the elegant solution to multi-armed bandits and the basis of online recommendation and A/B testing, maintains a Beta posterior per arm and updates it with one addition per observation. Bayesian optimization, Kalman filters (Normal–Normal updates in disguise), and latent Dirichlet allocation for topic modeling all lean on conjugacy for fast, closed-form belief updates.

from scipy.stats import beta, gamma
# Beta-Binomial: prior Beta(2,2), observe 8 heads in 10
a, b = 2 + 8, 2 + (10 - 8)
print("Beta-Binomial posterior mean:", round(a/(a+b), 3))     # 0.571
# Gamma-Poisson: prior Gamma(2, 1), observe 30 events over 10 periods
shape, rate = 2 + 30, 1 + 10
print("Gamma-Poisson posterior mean rate:", round(shape/rate, 3))  # 2.909

Chapter summary

  • A conjugate prior yields a posterior in the same family, so updating is parameter arithmetic — no integral.

  • Beta–Binomial: \(\Betad(\alpha,\beta)\to\Betad(\alpha+h,\beta+n-h)\); \(\alpha,\beta\) are pseudo-counts; flat prior gives Laplace’s rule.

  • Gamma–Poisson: \(\Gammad(\alpha,\beta)\to\Gammad(\alpha+\sum x,\beta+n)\) for rates.

  • Normal–Normal: precisions add; the posterior mean is a precision-weighted average; SD shrinks like \(1/\sqrt n\).

  • Conjugacy follows from the exponential family; it is convenient but limited, motivating MCMC/VI. It underlies Naive Bayes smoothing, Thompson sampling, and Kalman filters.