A point estimate alone is dangerous: it hides its own uncertainty. This chapter adds the error bars. Confidence intervals express how precise an estimate is; hypothesis tests decide whether an observed effect is real or could be chance. These frequentist tools are how we compare models honestly, report results responsibly, and avoid being fooled by noise — including the noise in our own benchmark numbers.

Sampling distributions and the standard error

An estimator like the sample mean has a sampling distribution; by the CLT (Chapter [ch:limit]) it is approximately Gaussian for large \(n\), centered at the true value with standard error \[\SE(\bar X)=\frac{\sigma}{\sqrt n}\quad(\text{estimated by }s/\sqrt n).\] The standard error is the typical distance between estimate and truth — and it shrinks only as \(1/\sqrt n\).

Confidence intervals

A \(95\%\) confidence interval is a data-computed range produced by a procedure that, over repeated samples, contains the true parameter \(95\%\) of the time. For a mean (large \(n\)): \[\bar X \pm 1.96\,\frac{s}{\sqrt n}.\]

The correct interpretation is subtle and constantly mangled. A \(95\%\) CI does not mean “there is a \(95\%\) probability the true value is in this interval” — in the frequentist view the true value is fixed and the interval is random, so a specific interval either contains it or not. The \(95\%\) refers to the long-run success rate of the procedure. (The statement you want to make — probability over the parameter — is exactly what a Bayesian credible interval provides, Chapter [ch:bayesian].)

A confidence interval is the estimate plus an honest margin of error. Wider intervals mean less certainty (small \(n\), high variance); narrower means more. When two models’ performance intervals overlap heavily, the apparent winner may just be lucky — which is why serious benchmarking reports intervals, not bare numbers.

Hypothesis testing

A hypothesis test pits a null hypothesis \(H_0\) (“no effect,” the skeptical default) against an alternative \(H_1\). We compute a test statistic from the data and ask how extreme it would be if \(H_0\) were true.

The \(p\)-value is the probability, assuming \(H_0\) is true, of observing data at least as extreme as what we saw. A small \(p\)-value means the data would be surprising under \(H_0\), casting doubt on it. If \(p<\alpha\) (commonly \(0.05\)), we “reject \(H_0\).”

The \(p\)-value is the most misinterpreted number in science. It is not the probability that \(H_0\) is true, nor the probability your result is a fluke, nor one minus the probability of replication. It is \(P(\text{data this extreme}\mid H_0)\) — a statement about the data given the hypothesis, not the hypothesis given the data. Confusing the two is, once again, the inversion that only Bayes’ theorem (with a prior) can legitimately perform. Beware also \(p\)-hacking: test enough things and some will cross \(0.05\) by chance.

Errors, power, and multiple testing

Every test risks two errors:

\(H_0\) true \(H_0\) false
Reject \(H_0\) Type I error (rate \(\alpha\)) correct (power \(=1-\beta\))
Fail to reject correct Type II error (rate \(\beta\))

Power is the chance of detecting a real effect; it grows with sample size and effect size. Common tests include the \(z\)- and \(t\)-tests (means; the \(t\) handles unknown variance and small samples via its heavy tails), the \(\chi^2\) test (categorical counts), and ANOVA (several groups). When running many tests, correct for multiple comparisons (e.g. Bonferroni) or the false-discovery rate, or chance alone will manufacture “significant” findings.

Resampling: the bootstrap

When formulas are hard, bootstrapping estimates uncertainty by resampling the data with replacement many times and recomputing the statistic. The spread of those replicates approximates the sampling distribution — a computational shortcut to standard errors and confidence intervals for almost any statistic, and a close cousin of the bagging/ensemble idea in ML.

These tools are how ML practitioners avoid fooling themselves. A model that scores \(91.2\%\) vs. a baseline’s \(90.8\%\) has likely won nothing if the confidence interval on the gap straddles zero — benchmark leaderboards demand error bars and significance tests for exactly this reason. A/B testing of deployed models is hypothesis testing. Bootstrapping produces confidence intervals on metrics like AUC and powers bagging and random forests. Power analysis sizes experiments. And the \(p\)-value pitfalls reappear as the replication and “SOTA-chasing” crises in ML research, where multiple-comparison and selection effects inflate apparent gains.

import numpy as np
from scipy import stats
# 95% CI for a mean and a two-sample t-test
a = np.random.normal(0.0, 1.0, 200)
b = np.random.normal(0.3, 1.0, 200)
se = a.std(ddof=1)/np.sqrt(len(a))
print((a.mean()-1.96*se, a.mean()+1.96*se))      # ~ CI around 0
t, p = stats.ttest_ind(a, b)
print(round(t,2), round(p,4))                     # small p => means differ

# Bootstrap CI for the median (no formula needed)
x = np.random.exponential(1.0, 500)
boot = [np.median(np.random.choice(x, len(x), replace=True)) for _ in range(5000)]
print(np.percentile(boot, [2.5, 97.5]).round(3))  # 95% bootstrap interval

Chapter summary

  • The standard error \(\sigma/\sqrt n\) is the typical estimate–truth distance; it shrinks only as \(1/\sqrt n\).

  • A confidence interval reflects the long-run coverage of a procedure, not the probability that this interval contains the truth.

  • A \(p\)-value is \(P(\text{data this extreme}\mid H_0)\) — not \(P(H_0\mid\text{data})\); mind Type I/II errors, power, and multiple testing.

  • Use \(z\)/\(t\)/\(\chi^2\)/ANOVA as appropriate; bootstrap for uncertainty without formulas.

  • In ML these give honest model comparison, A/B tests, metric confidence intervals, and the basis of bagging.