Before any formula, Bayesian inference is a way of reasoning: hold beliefs as probabilities, and revise them in proportion to the evidence. This chapter sets up that worldview — probability as degree of belief, the contrast with the frequentist view, and the single update rule that the rest of the book makes precise — so that every later technique reads as a variation on one natural idea.

Two meanings of “probability”

There are two dominant interpretations of probability:

  • Frequentist: a probability is the long-run frequency of an event in repeated trials. “The coin is fair” means it lands heads 50% of the time over infinitely many flips.

  • Bayesian: a probability is a degree of belief about a proposition, given what you know. “There is a 70% chance it will rain tomorrow” or “a 95% chance this parameter lies between 2 and 4” — statements about one-off events and unknown quantities.

The Bayesian view lets you assign probabilities to hypotheses and parameters, not just to repeatable events. This is the move that makes everything else possible: if a parameter can have a probability distribution, then data can update that distribution. “What is the probability this model is right?” is a meaningful Bayesian question and a forbidden frequentist one.

Learning as belief revision

You already think like a Bayesian. You hear a strange noise at night; your prior says “probably the wind.” You hear it again, rhythmic and close; the evidence shifts your belief toward “someone is at the door.” You did not discard your prior — you combined it with new data. Bayesian inference formalizes exactly this: belief after \(=\) belief before, reweighted by how well each possibility predicts what you observed.

As evidence accumulates the belief curve marches toward the truth and narrows — uncertainty shrinking as data grows. That sharpening is the visual signature of Bayesian learning, and we will see it again and again.

The update rule, in words

Every Bayesian analysis has the same three ingredients and one rule: \[\underbrace{P(\Hyp\mid D)}_{\text{posterior}}\ \propto\ \underbrace{P(D\mid \Hyp)}_{\text{likelihood}}\ \times\ \underbrace{P(\Hyp)}_{\text{prior}}.\] Prior: what you believed about hypothesis \(\Hyp\) before. Likelihood: how well \(\Hyp\) predicts the data \(D\). Posterior: your updated belief. The posterior is always a compromise between prior and likelihood — and as data accumulates, the likelihood dominates. Chapter [ch:theorem] derives this; the whole book is its consequences.

Why think this way?

  • Uncertainty is first-class. The answer is a distribution, so you always know how sure you are — crucial for risk, science, and safety-critical AI.

  • Prior knowledge is usable. You can fold in physics, past studies, or expert judgment instead of starting from scratch.

  • It is coherent. Under mild axioms (Cox, de Finetti), degree-of-belief reasoning must obey the probability rules — otherwise a clever bettor can guarantee a profit against you (a “Dutch book”).

  • It updates sequentially. Today’s posterior is tomorrow’s prior, so learning is naturally online and incremental.

Bayesian reasoning is powerful but not magic. The conclusions depend on the prior you choose, so a poorly justified prior can bias results — you must be explicit and check sensitivity (Chapter [ch:priors]). And exact computation is often intractable, which is why much of the field (Chapter [ch:mcmc]) is about approximating the posterior. “Subjective prior” is a feature, not a bug, but it demands honesty.

This worldview is the backbone of probabilistic machine learning. Treating model parameters as random variables with priors gives Bayesian neural networks and Gaussian processes; the prior-as-belief idea is exactly regularization (Chapter [ch:bayesmlai]); and “answers are distributions” is the foundation of uncertainty quantification, without which a model cannot say “I don’t know.” Sequential updating underlies online learning, Bayesian optimization, and reinforcement learning. Even when a system is not explicitly Bayesian, the Bayesian lens explains why it works.

import numpy as np
from scipy.stats import beta
# Belief about a coin's bias theta, updated as flips arrive (Beta-Binomial, Ch.4)
a, b = 1, 1                      # flat prior: all biases equally plausible
flips = "HHTHHHTHHH"            # observed data
for f in flips:
    a += (f == "H"); b += (f == "T")
    lo, hi = beta.ppf([0.025, 0.975], a, b)
    print(f"after {f}: mean={a/(a+b):.2f}  95% CI=[{lo:.2f},{hi:.2f}]")
# Belief shifts toward heads-bias and the interval tightens with more data

Chapter summary

  • Frequentist probability is long-run frequency; Bayesian probability is degree of belief — and can be assigned to hypotheses and parameters.

  • Learning is belief revision: combine prior belief with the likelihood of the data to get a posterior.

  • The master rule is posterior \(\propto\) likelihood \(\times\) prior; the posterior is a compromise that tilts toward the data as it accumulates.

  • Benefits: first-class uncertainty, usable prior knowledge, coherence, and sequential updating.

  • Caveats: results depend on the prior, and exact computation is often intractable — the motivations for Chapters [ch:priors] and [ch:mcmc].