Conjugacy is a luxury; most real posteriors have no closed form because the evidence integral \(p(D)=\int p(D\mid\theta)p(\theta)\,d\theta\) is intractable. The modern Bayesian toolkit sidesteps it two ways: MCMC draws samples from the posterior without ever computing the normalizer, and variational inference replaces sampling with optimization. Together they unlocked Bayesian methods for complex models and made Bayesian deep learning possible.

The intractable normalizer

We can almost always evaluate the unnormalized posterior \(\tilde p(\theta)=p(D\mid\theta)p(\theta)\) — it is just likelihood times prior. What we cannot do is compute the constant \(p(D)\) that makes it integrate to one, because that integral is high-dimensional and has no closed form. Every computational Bayesian method is a way to answer questions about the posterior using only the unnormalized \(\tilde p\). That is the central trick of the entire field.

Markov chain Monte Carlo

MCMC constructs a Markov chain whose stationary distribution is the posterior. Running the chain and collecting its states yields (correlated) samples from \(p(\theta\mid D)\), from which any expectation, mean, or interval is estimated by averaging.

Propose a move \(\theta\to\theta'\); accept with probability \(\min\!\big(1,\frac{\tilde p(\theta')q(\theta\mid\theta')}{\tilde p(\theta)q(\theta'\mid\theta)}\big)\). The normalizer cancels in the ratio — which is why we never need it. Always go uphill; sometimes go downhill, so the chain explores.

When full conditionals \(p(\theta_i\mid\theta_{-i},D)\) are known, sample each parameter in turn from its conditional. No tuning, ideal for conjugate substructure.

Use the gradient of \(\log\tilde p\) to propose distant, high-acceptance moves via simulated physical dynamics. The state of the art for continuous parameters; the engine inside Stan and PyMC.

MCMC is exact in the limit but tricky in practice. Discard early burn-in samples (before the chain reaches the posterior). Check convergence — run multiple chains and compare with the \(\hat R\) statistic (\(\hat R\approx 1\) is good), inspect trace plots, and watch effective sample size (correlated draws carry less information). A chain stuck in one mode of a multimodal posterior can look perfectly converged while being completely wrong. Never trust a single short chain.

Variational inference

Variational inference (VI) turns inference into optimization: pick a tractable family \(q_\phi(\theta)\) (e.g. Gaussians) and find the member closest to the true posterior by minimizing the KL divergence \(\KL(q_\phi\,\|\,p(\cdot\mid D))\). Since we cannot evaluate that KL directly, we equivalently maximize the evidence lower bound (ELBO): \[\mathrm{ELBO}(\phi)=\E_{q_\phi}\big[\log p(D,\theta)\big]-\E_{q_\phi}\big[\log q_\phi(\theta)\big]\ \le\ \log p(D).\]

The ELBO splits into “fit the data well” minus “stay spread out” (an entropy term) — a built-in Occam’s razor. VI is typically much faster than MCMC and scales to huge models and datasets via stochastic gradients, which is why it dominates Bayesian deep learning. The cost: \(q\) is an approximation, and the usual mean-field \(q\) tends to underestimate posterior variance (it finds one mode and hugs it), making VI overconfident relative to MCMC.

These algorithms are what make Bayesian deep learning real. Bayes by Backprop trains a Bayesian neural network by maximizing the ELBO over weight distributions; MC dropout reinterprets dropout at test time as variational inference, giving cheap uncertainty from an ordinary net. Stochastic-gradient MCMC (SGLD, SG-HMC) scales sampling to mini-batches and big models. The reparameterization trick that makes the ELBO differentiable is the same one that trains variational autoencoders. Probabilistic programming systems (Stan, PyMC, Pyro, NumPyro) provide NUTS and VI as push-button inference, so practitioners specify a model and let these algorithms compute the posterior.

import numpy as np
rng = np.random.default_rng(0)
data = rng.normal(2.0, 1.0, size=50)
def log_post(mu):                     # unnormalized: Normal likelihood + N(0,10^2) prior
    return -0.5*np.sum((data-mu)**2) - 0.5*(mu/10)**2
# Metropolis sampler -- note we only ever use the UNNORMALIZED log-posterior
mu, samples = 0.0, []
for _ in range(20000):
    prop = mu + rng.normal(0, 0.3)
    if np.log(rng.uniform()) < log_post(prop) - log_post(mu):
        mu = prop
    samples.append(mu)
s = np.array(samples[2000:])          # discard burn-in
print("posterior mean ~", round(s.mean(), 3), " 95% CrI", np.round(np.percentile(s,[2.5,97.5]),3))

Chapter summary

  • Posteriors are usually intractable because the evidence integral has no closed form; methods use only the unnormalized posterior.

  • MCMC builds a Markov chain whose stationary distribution is the posterior: Metropolis–Hastings, Gibbs, and gradient-based HMC/NUTS.

  • Diagnose MCMC with burn-in, multiple chains, \(\hat R\), and effective sample size; beware multimodality.

  • Variational inference maximizes the ELBO to fit a tractable \(q\) — fast and scalable but typically underestimates variance.

  • These power Bayes by Backprop, MC dropout, SG-MCMC, VAEs, and probabilistic programming — the machinery of Bayesian deep learning.