The prior is what makes Bayesian inference both powerful and controversial: it lets you inject knowledge, but it also forces you to be explicit about your assumptions. This chapter is a practical guide to choosing priors — informative, weakly informative, and “non-informative” — and to the honest practice of checking that your conclusions do not hinge on an arbitrary choice.

A spectrum of priors

Priors range from strongly opinionated to deliberately vague:

  • Informative: encodes real knowledge (past studies, physics, expert elicitation). Narrow; strongly shapes the posterior when data is limited.

  • Weakly informative: rules out the absurd (e.g. a human height is not 5 m) but stays open. The modern default — it regularizes gently without dominating.

  • Non-informative / reference: aims to “let the data speak” (flat, Jeffreys). Useful as a baseline, but trickier than it looks.

There is no such thing as no prior — choosing to do frequentist maximum likelihood is itself the choice of a flat prior. The real question is not “prior or no prior?” but “which prior, and how much does it matter?” A good default is a weakly informative prior: confident enough to stabilize estimates and prevent nonsense, humble enough to be overruled by data.

Non-informative priors and their subtleties

A flat prior (\(p(\theta)\propto 1\)) looks neutral but is not: flatness is not preserved under reparameterization, so “uniform on \(\theta\)” is informative about \(\theta^2\) or \(\log\theta\). The Jeffreys prior, \(p(\theta)\propto\sqrt{I(\theta)}\) (the square root of Fisher information), fixes this by being invariant to how you parameterize — for a Binomial probability it is \(\Betad(\tfrac12,\tfrac12)\), gently favoring the extremes. Such priors can also be improper (not integrate to one); that is acceptable only if the resulting posterior is proper.

Priors as regularization

A prior pulls estimates toward sensible values — which is exactly what regularization does. The posterior-mode (MAP) estimate is \[\hat\theta_{\text{MAP}}=\argmax_\theta\ \big[\log p(D\mid\theta)+\log p(\theta)\big],\] i.e. log-likelihood plus a penalty \(\log p(\theta)\). A Gaussian prior gives the \(L_2\) penalty (ridge / weight decay); a Laplace prior gives the \(L_1\) penalty (lasso / sparsity). Every penalty you add to a loss is, secretly, a prior belief about the parameters.

Eliciting and stress-testing priors

A practical workflow:

  1. Put parameters on an interpretable scale (standardize predictors) so a generic prior like \(\Normal(0,1)\) is meaningful.

  2. Use prior predictive checks: simulate fake data from the prior; if it produces impossible datasets, the prior is wrong.

  3. Run a sensitivity analysis: refit with a few different reasonable priors and confirm the conclusions are stable.

Two opposite errors. (1) An overconfident prior (too narrow, centered wrong) can stubbornly override the data and bias the posterior — dangerous when you do not actually have that much prior knowledge. (2) A vague/flat prior is not automatically “objective”: it can put most of its mass on absurd values, cause improper posteriors, or behave badly under transformation. When a result changes drastically with a slightly different reasonable prior and little data, report that fragility rather than hiding it.

The prior-as-regularizer equivalence is one of the most useful bridges between Bayesian statistics and deep learning. Weight decay is a zero-mean Gaussian prior on the weights; \(L_1\) regularization is a Laplace prior driving sparsity; early stopping and dropout act as implicit priors. In Bayesian neural networks the choice of weight prior (Gaussian, heavy-tailed Student-\(t\), or function-space priors) is an active research area precisely because, with millions of parameters and finite data, the prior still shapes generalization. Understanding priors lets you read regularization choices as statements of belief.

import numpy as np
from scipy.stats import beta
h, n = 2, 3                          # tiny dataset: prior will matter a lot
grid = np.linspace(0, 1, 1001)
for (a, b, name) in [(1,1,"flat"), (0.5,0.5,"Jeffreys"), (20,20,"strong=fair")]:
    post = beta(a+h, b+n-h)
    print(f"{name:>12} prior -> posterior mean {post.mean():.3f}")
# Conclusions differ with little data -> always do this sensitivity check

Chapter summary

  • Priors span informative, weakly informative (the modern default), and non-informative/reference.

  • There is always a prior; flat priors are not truly neutral (not reparameterization-invariant), which motivates the Jeffreys prior.

  • MAP \(=\) penalized likelihood: Gaussian prior \(\to\) \(L_2\)/ridge, Laplace prior \(\to\) \(L_1\)/lasso — penalties are priors.

  • Elicit on interpretable scales; use prior predictive checks and sensitivity analysis.

  • Beware overconfident priors (bias) and vague priors (absurd mass, impropriety); report fragility honestly.