Why does averaging more data give a better answer? Why are so many things bell-shaped? Why can we estimate an expectation by a sample mean — the move that justifies mini-batch training? The limit theorems answer these questions. They are the theoretical bridge between probability (the model) and statistics (the data), and they are the reason machine learning works at all.

Probability inequalities

Before the limit theorems, two inequalities let us bound probabilities knowing only a mean or variance.

For a nonnegative random variable and any \(a>0\): \(\;P(X\ge a)\le \dfrac{\E[X]}{a}\) (Markov). For any variable with finite variance: \(\;P(|X-\mu|\ge k\sigma)\le \dfrac{1}{k^2}\) (Chebyshev).

These are crude but assumption-light: Chebyshev says no distribution can put more than \(1/k^2\) of its mass beyond \(k\) standard deviations — at most \(25\%\) past \(2\sigma\), for any shape whatsoever. Sharper “concentration inequalities” (Hoeffding, Bernstein) tighten this for sums of bounded variables and are the backbone of generalization bounds in learning theory: they bound how far training error can stray from true error.

For a convex function \(\varphi\), \(\;\varphi(\E[X])\le \E[\varphi(X)]\). (Reversed for concave \(\varphi\).)

Jensen is the workhorse behind the ELBO in variational inference and the non-negativity of KL divergence (Chapter [ch:information]).

The Law of Large Numbers

Let \(X_1,X_2,\dots\) be i.i.d. with mean \(\mu\). The sample mean \(\bar X_n=\tfrac1n\sum_{i=1}^n X_i\) converges to \(\mu\) as \(n\to\infty\): \[\bar X_n \longrightarrow \mu.\]

The LLN is the bedrock guarantee of statistics: averages of more data converge to the truth. It is what makes the frequentist interpretation coherent (long-run frequency \(\to\) probability) and what licenses Monte Carlo estimation — approximating an expectation \(\E[g(X)]\) by a sample average \(\tfrac1N\sum g(x_i)\). Every time we estimate a loss, a gradient, or an integral by averaging samples, we are leaning on the LLN.

The Central Limit Theorem

The LLN says the sample mean converges; the CLT says how — and reveals why the Gaussian is universal.

For i.i.d. \(X_i\) with mean \(\mu\) and finite variance \(\sigma^2\), the standardized sample mean converges to a standard normal: \[\frac{\bar X_n-\mu}{\sigma/\sqrt{n}}\ \longrightarrow\ \Normal(0,1).\] Equivalently, \(\bar X_n\) is approximately \(\Normal\!\big(\mu,\,\sigma^2/n\big)\) for large \(n\).

The CLT is one of the most astonishing facts in mathematics: sum up many independent random effects and the result is bell-shaped, no matter what the individual effects look like — uniform, skewed, discrete, anything (with finite variance). This is why measurement errors, heights, and averaged quantities are Gaussian. Note the \(\sqrt{n}\): the standard error of the mean shrinks like \(1/\sqrt{n}\), so to halve your error you need four times the data — the fundamental, sometimes painful, exchange rate of statistics.

Modes of convergence (brief)

Probabilists distinguish convergence in probability (the LLN), in distribution (the CLT), and almost surely (the strong LLN). The distinctions matter for proofs; for our purposes the headline is enough: sample averages converge to expectations, and their fluctuations are Gaussian of size \(\sigma/\sqrt n\).

The limit theorems are the silent justification of everyday ML. Mini-batch SGD replaces the true gradient (an expectation over all data) with an average over a batch — valid by the LLN, with noise of order \(1/\sqrt{\text{batch size}}\) by the CLT, which is why larger batches give smoother, lower-variance gradient estimates. Monte Carlo estimates of integrals (the reparameterization trick, dropout-as-Bayesian-inference, RL returns) converge by the LLN. Confidence intervals and error bars on metrics use the CLT’s \(\sigma/\sqrt n\). Concentration inequalities give the generalization bounds of statistical learning theory. And the CLT explains why Gaussian assumptions are so often safe: averaged quantities are bell-shaped by law.

import numpy as np
# LLN: sample mean of a skewed (exponential) variable -> true mean 1.0
x = np.random.exponential(scale=1.0, size=2_000_000)
print(round(x.mean(), 3))                    # ~1.0

# CLT: averages of n exponential draws look Gaussian; SE shrinks like 1/sqrt(n)
for n in (1, 10, 100, 1000):
    means = np.random.exponential(1.0, size=(100_000, n)).mean(axis=1)
    print(n, round(means.std(), 4))          # ~ 1/sqrt(n): 1, 0.316, 0.1, 0.0316

Chapter summary

  • Markov/Chebyshev bound tail probabilities from a mean/variance alone; concentration inequalities sharpen this for generalization bounds.

  • Jensen’s inequality (\(\varphi(\E X)\le\E[\varphi(X)]\) for convex \(\varphi\)) underlies the ELBO and \(\KL\ge0\).

  • The Law of Large Numbers: sample averages converge to the true mean — the basis of Monte Carlo and mini-batch estimation.

  • The Central Limit Theorem: standardized sample means become \(\Normal(0,1)\); the standard error shrinks like \(\sigma/\sqrt n\).

  • Together they justify SGD’s batch gradients, Monte Carlo objectives, and confidence intervals.