The two great schools of statistics answer the same questions in fundamentally different ways. Understanding the contrast sharpens your grasp of both — and clears up the most persistent confusions in applied statistics, like what a confidence interval and a \(p\)-value actually mean. This chapter compares the philosophies, the interval estimates, and the approaches to hypothesis testing, then argues that the practical question is “which tool for which job?”
Two philosophies
The split is about what is random:
Frequentist: parameters are fixed unknown constants; data is random. Probability describes the long-run behavior of procedures over hypothetical repeated samples.
Bayesian: parameters are random (we model our uncertainty about them); the observed data is fixed. Probability describes degree of belief.
Confidence vs. credible intervals
This is the most misunderstood pair in statistics.
A 95% confidence interval (frequentist): if we repeated the experiment many times, 95% of the intervals so constructed would contain the true \(\theta\). The probability is a property of the procedure, not of your one interval — which either contains \(\theta\) or does not.
A 95% credible interval (Bayesian): given the data and prior, there is a 95% probability that \(\theta\) lies in this interval — a direct statement about the parameter.
People almost always want the credible interpretation; only the Bayesian framework licenses it.
Hypothesis testing
Frequentists test a null hypothesis \(H_0\) via a \(p\)-value: \(P(\text{data at least this extreme}\mid H_0)\). Small \(p\) “rejects” \(H_0\). Bayesians compare hypotheses with the Bayes factor, the ratio of evidences: \[\mathrm{BF}_{10}=\frac{P(D\mid H_1)}{P(D\mid H_0)},\] which multiplies the prior odds into the posterior odds: \(\frac{P(H_1\mid D)}{P(H_0\mid D)}=\mathrm{BF}_{10}\cdot\frac{P(H_1)}{P(H_0)}\).
The \(p\)-value is not the probability that \(H_0\) is true, nor the probability your results are due to chance. It is \(P(\text{data}\mid H_0)\), not \(P(H_0\mid\text{data})\) — confusing the two is the inverse fallacy of Chapter [ch:theorem]. A \(p\)-value also says nothing about effect size, depends on the (often unstated) sampling intention, and is rampantly abused through “\(p\)-hacking.” The Bayes factor answers the question people actually ask — “how much should I believe \(H_1\) over \(H_0\)?” — but it depends on the prior and can be sensitive to it (the Jeffreys–Lindley paradox).
When they agree, and when they differ
With lots of data and a weak prior, Bayesian and frequentist answers usually numerically coincide — the likelihood dominates, and a credible interval matches a confidence interval. They diverge when (a) data is scarce, where the prior matters; (b) prior information is genuinely available and valuable; or (c) the question is inherently about belief in a hypothesis or about a one-off event. The frameworks are complementary lenses, not rivals to be ranked once and for all.
| Frequentist | Bayesian | |
|---|---|---|
| Parameter | fixed constant | random (belief) |
| Needs a prior? | no | yes |
| Interval | confidence (procedure) | credible (parameter) |
| Test tool | \(p\)-value | Bayes factor / posterior |
| Strength | no prior, fast, calibrated | uncertainty, prior info, intuitive |
| Cost | misinterpreted, no prior info | prior choice, computation |
Machine learning is thoroughly pragmatic and uses both. Plain maximum likelihood training and cross-validation are frequentist in spirit; regularization, Bayesian neural networks, Gaussian processes, and Bayesian optimization are Bayesian. The most useful stance is to know which question you are asking: “what procedure has good average performance?” (frequentist, e.g. test-set accuracy and CV) versus “how uncertain am I about this prediction?” (Bayesian, e.g. predictive posteriors and calibration). Modern probabilistic ML freely mixes them — e.g. empirical Bayes and frequentist evaluation of Bayesian models.
import numpy as np
from scipy import stats
rng = np.random.default_rng(0)
data = rng.normal(1.0, 2.0, size=30)
# Frequentist 95% confidence interval for the mean
m, se = data.mean(), stats.sem(data)
ci = stats.t.interval(0.95, len(data)-1, loc=m, scale=se)
print("frequentist 95% CI:", np.round(ci, 3))
# Bayesian 95% credible interval (Normal-Normal, vague prior) ~ coincides here
tau = 1/(1/1e6 + len(data)/data.var()); mu = tau*(len(data)*m/data.var())
print("bayesian 95% CrI:", np.round(stats.norm(mu, tau**0.5).ppf([.025,.975]), 3))Chapter summary
Frequentist: parameters fixed, data random, no prior; Bayesian: parameters random, data fixed, prior required.
A confidence interval is a statement about the procedure’s long-run coverage; a credible interval is a probability statement about the parameter.
\(p\)-value \(=P(\text{data}\mid H_0)\), not \(P(H_0\mid\text{data})\); the Bayes factor compares evidence for hypotheses but depends on priors.
With ample data and weak priors the two usually agree; they diverge when data is scarce or prior information matters.
ML uses both pragmatically: MLE/CV (frequentist) and regularization/BNNs/GPs (Bayesian) — match the tool to the question.