Inference within a model assumes the model is right. But which model should we use? Bayesian model comparison answers this with the very quantity we worked so hard to avoid — the evidence \(p(D)\) — which turns out to automatically embody Occam’s razor, preferring models that are no more complex than the data demands. This chapter covers the evidence, Bayes factors, practical predictive criteria, and the indispensable habit of checking models against reality.

The evidence as a model score

For a model \(M\), the marginal likelihood (evidence) is the probability it assigns to the data, averaged over its own prior: \[p(D\mid M)=\int p(D\mid\theta,M)\,p(\theta\mid M)\,d\theta.\] This is the normalizer from Bayes’ theorem — ignorable within a model, decisive between models.

The evidence rewards models that predict the observed data well before seeing it — averaged over all parameter settings the prior allows. A model too simple cannot fit the data (low likelihood everywhere); a model too complex spreads its prior probability thinly over many datasets it could have produced, so it assigns little to the one we actually saw. The evidence peaks at the model of just enough complexity — Occam’s razor falls out of the mathematics, with no ad hoc penalty added.

Bayes factors

The Bayes factor comparing models \(M_1,M_0\) is the ratio of their evidences, \[\mathrm{BF}_{10}=\frac{p(D\mid M_1)}{p(D\mid M_0)},\] which updates the prior odds on the models into posterior odds. Conventionally \(\mathrm{BF}>10\) is strong, \(>100\) decisive evidence for \(M_1\).

The evidence is powerful but treacherous. It is sensitive to the prior: unlike the posterior, \(p(D\mid M)\) does not wash out a vague prior even with infinite data, so a carelessly diffuse prior can arbitrarily penalize a model (the Jeffreys–Lindley paradox). It is also hard to compute — that intractable integral again — and naive estimators (like the harmonic-mean estimator) are notoriously unreliable. For these reasons many practitioners prefer predictive criteria below for model selection, reserving Bayes factors for carefully specified hypotheses.

Predictive criteria: WAIC and cross-validation

A robust, practical alternative is to score models by their out-of-sample predictive accuracy, estimated without a separate test set:

  • WAIC (widely applicable information criterion): uses the full posterior predictive and an effective-parameter penalty — a fully Bayesian generalization of AIC.

  • PSIS-LOO: approximate leave-one-out cross-validation computed cheaply from MCMC samples via importance sampling.

These target predictive performance directly, are far less prior-sensitive than the evidence, and come with diagnostics — the recommended default for everyday model comparison.

Posterior predictive checks

Before comparing models, ask whether a model is adequate at all. A posterior predictive check simulates datasets from the fitted model and compares them to the real data: if the model cannot reproduce key features (the spread, the tails, the number of zeros), it is wrong regardless of its evidence. “All models are wrong” — predictive checks tell you whether yours is useful, and they catch failures that any single summary score hides.

There is a deep alternative to selection: model averaging. Rather than pick one model, weight each model’s predictions by its posterior probability and average. This propagates model uncertainty into predictions and often beats any single model — the Bayesian justification for why ensembles work so well in machine learning.

These ideas reappear throughout ML, sometimes in disguise. The evidence is the basis of empirical Bayes and of Gaussian process hyperparameter tuning (maximizing the marginal likelihood, a.k.a. type-II maximum likelihood or “the evidence framework”), and of the evidence lower bound (ELBO) that trains VAEs and Bayesian neural networks — the ELBO is a tractable surrogate for \(\log p(D)\). Occam’s-razor-by-evidence explains why Bayesian methods resist overfitting. Bayesian model averaging is the principled story behind ensembles and deep ensembles, and predictive scoring (LOO/WAIC) mirrors the cross-validation at the heart of model selection in ML.

import numpy as np
# Evidence (marginal likelihood) via simple Monte Carlo over the prior:
# does a quadratic beat a line on curved data?
rng = np.random.default_rng(0)
x = np.linspace(-2, 2, 25); y = 0.8*x**2 + rng.normal(0, 0.5, 25)
def evidence(degree, S=20000):
    coefs = rng.normal(0, 3, size=(S, degree+1))    # samples from the prior
    Phi = np.vstack([x**k for k in range(degree+1)]).T
    pred = coefs @ Phi.T
    loglik = -0.5*((y - pred)**2).sum(1)/0.5**2      # Gaussian likelihood
    return np.log(np.mean(np.exp(loglik - loglik.max()))) + loglik.max()
print("log evidence  line:", round(evidence(1), 1),
      " quadratic:", round(evidence(2), 1))          # quadratic wins on curved data

Chapter summary

  • The evidence \(p(D\mid M)\) scores a model; it automatically embodies Occam’s razor, favoring just-enough complexity.

  • Bayes factors compare evidences but are prior-sensitive (Jeffreys–Lindley) and hard to compute.

  • Prefer predictive criteria — WAIC, PSIS-LOO — for robust, low-prior-sensitivity model selection.

  • Always run posterior predictive checks: can the model reproduce the data? Model averaging beats selection and justifies ensembles.

  • The evidence underlies empirical Bayes, GP hyperparameter tuning, and the ELBO; these explain Bayesian resistance to overfitting.