Here is where everything converges. Machine learning is applied probability and statistics at scale: it posits a probabilistic model of data, estimates its parameters by (penalized) maximum likelihood, quantifies uncertainty, and generalizes from a sample to a population. Linear algebra and calculus give us the tools to compute; probability and statistics give us the reason those computations are the right ones. This chapter assembles the whole picture and shows that the training of any model — from a coin estimator to a diffusion transformer — is one statistical idea repeated.
Learning is statistical inference
A model defines a probability \(p(\vy\given\vx,\vtheta)\) of outputs given inputs and parameters. Training estimates \(\vtheta\) from data \(\Ldata\); prediction marginalizes or maximizes over outputs. The two estimation philosophies of this book map directly onto two views of training: \[\hat\vtheta_{\text{MLE}}=\argmax_{\vtheta}\, p(\Ldata\given\vtheta) \qquad\text{vs.}\qquad p(\vtheta\given\Ldata)\propto p(\Ldata\given\vtheta)\,p(\vtheta).\] The first is frequentist point estimation (most of deep learning); the second is the Bayesian posterior (uncertainty-aware ML). Both start from the same likelihood.
Where the loss functions come from
Every standard loss is a negative log-likelihood under some noise model (Chapters [ch:estimation], [ch:regression]):
| Probabilistic model | \(=\) Loss (NLL) | Used by |
|---|---|---|
| Gaussian targets | mean-squared error | regression, autoencoders |
| Bernoulli / Categorical | cross-entropy | classifiers, language models |
| Poisson counts | Poisson loss | count regression |
| Laplace noise | mean-absolute error | robust regression |
Choosing how you model the randomness chooses your loss function. Cross-entropy is not an arbitrary formula — it is the negative log-likelihood of a categorical model and equivalently the KL divergence from the model to the data (Chapter [ch:information]). Squared error is the NLL of Gaussian noise. So minimizing a loss is maximizing likelihood is minimizing a divergence to the true data distribution. This single thread ties Chapters [ch:estimation], [ch:regression], and [ch:information] together and runs underneath all supervised learning.
Regularization is a prior
Pure MLE overfits (the variance side of the bias–variance tradeoff). Real training adds a penalty, which Bayes reveals to be a prior (Chapter [ch:bayesian]): \[\underbrace{\min_{\vtheta}\ -\log p(\Ldata\given\vtheta)}_{\text{MLE / data fit}}\ +\ \underbrace{\lambda\,R(\vtheta)}_{-\log p(\vtheta)\ =\ \text{prior}}.\]
L2 / weight decay \(=\) Gaussian prior \(\vtheta\sim\Normal(0,\tau^2 I)\) (ridge).
L1 / lasso \(=\) Laplace prior (induces sparsity).
Dropout \(\approx\) Bayesian model averaging; label smoothing \(=\) a prior on labels; early stopping \(\approx\) implicit shrinkage.
Every regularizer is a statement of prior belief about the parameters.
Generalization: from sample to population
A model is trained on a sample but judged on the population — a purely statistical leap. The bias–variance tradeoff (Chapter [ch:regression]) and concentration inequalities (Chapter [ch:limit]) describe when it succeeds; cross-validation estimates population error; confidence intervals and significance tests (Chapter [ch:testing]) keep model comparisons honest. “This model generalizes” is a claim that training error is a good estimate of true risk — exactly the kind of claim statistics was built to evaluate.
Uncertainty and calibration
A trustworthy model knows what it does not know. Two kinds of uncertainty matter:
Aleatoric — irreducible noise in the data itself (the \(\sigma^2\) term).
Epistemic — reducible uncertainty about the parameters, captured by the posterior \(p(\vtheta\given\Ldata)\).
A model is calibrated if its stated confidence matches reality: among predictions made with \(80\%\) confidence, about \(80\%\) should be correct. Deep networks are notoriously overconfident; we measure the gap with Expected Calibration Error and fix it with temperature scaling, Bayesian neural networks, MC dropout, and deep ensembles — all of which approximate the Bayesian posterior predictive (Chapter [ch:bayesian]) to turn raw scores into honest probabilities. In medicine, autonomy, and finance, calibration matters as much as accuracy.
Generative models: learning the whole distribution
Discriminative models learn \(p(\vy\given\vx)\); generative models learn the full data distribution \(p(\vx)\) and sample from it. Each major family is a probabilistic construction from this book:
VAEs: maximize the ELBO — expected log-likelihood minus a KL to a Gaussian latent prior (Chapters [ch:bayesian], [ch:information]); trained with the reparameterization trick.
Diffusion models: a Markov chain (Chapter [ch:markov]) of Gaussian noising, reversed by a learned chain; the objective is a sum of KL/score-matching terms.
Autoregressive models (GPT-style): factor \(p(\vx)=\prod_t p(x_t\given x_{<t})\) by the chain rule (Chapter [ch:bayes]) and fit each factor by cross-entropy.
GANs: a two-player game whose equilibrium minimizes a divergence between real and generated distributions.
The grand summary table
| Probability / statistics concept | Ch. | Where it powers AI |
|---|---|---|
| Probability axioms; events | [ch:whatis] | valid model outputs; softmax as a distribution |
| Conditional prob. & chain rule | [ch:bayes] | autoregressive factorization \(p(x)=\prod p(x_t\mid x_{<t})\) |
| Bayes’ theorem | [ch:bayes] | the template for learning; spam/diagnosis; priors |
| Random variables; expectation | [ch:randomvars] | risk \(=\E[\text{loss}]\); standardization / normalization |
| Distributions (zoo) | [ch:distributions] | choosing a model \(=\) choosing a loss |
| Covariance; multivariate Gaussian | [ch:joint] | PCA, whitening, GPs, VAE/diffusion priors |
| LLN & CLT | [ch:limit] | mini-batch SGD; Monte Carlo; error bars |
| Maximum likelihood | [ch:estimation] | cross-entropy and MSE losses |
| Bias–variance; CV | [ch:estimation],[ch:regression] | over/underfitting; capacity & regularization choices |
| Confidence & tests | [ch:testing] | honest benchmarking; A/B tests; bootstrapped metrics |
| Bayesian inference; MAP | [ch:bayesian] | weight decay (L2), lasso (L1), calibrated uncertainty |
| Regression / GLMs | [ch:regression] | linear/logistic/softmax \(=\) a net’s output layer |
| Markov chains & MCMC | [ch:markov] | RL/MDPs; posterior sampling; diffusion chains |
| Entropy / cross-entropy / KL / MI | [ch:information] | classification loss; ELBO; VI; contrastive learning |
Machine learning is the practice of inferring a probabilistic model from data and using it to predict under uncertainty. A model specifies a likelihood; maximum likelihood turns that into a loss (cross-entropy, squared error); a prior turns into a regularizer (weight decay, lasso); the bias–variance tradeoff and the limit theorems govern whether it generalizes; Bayes’ theorem and the posterior predictive supply calibrated uncertainty; Markov chains and Monte Carlo generate samples; and information theory measures the gap between model and world. Probability says what to expect, statistics says what to believe, and together they are not a prerequisite to machine learning — they are machine learning, with linear algebra and calculus as the tools that make the computations run.
Where to go next
With the companion Linear Algebra and Calculus volumes, you now hold the full mathematical core of machine learning: the vectors and matrices that represent data, the derivatives that train models, and the probability and statistics that justify the whole enterprise. The natural next steps are statistical learning theory (generalization bounds), the optimization of large models, and the deep-learning curriculum proper — where these distributions, likelihoods, and divergences come alive in classifiers, language models, and generative AI. You now understand why learning works; the rest is building.
Chapter summary
Learning is statistical inference: a likelihood model fit by (penalized) maximum likelihood, with prediction by marginalization/maximization.
Losses are negative log-likelihoods; regularizers are priors; both follow from the modeling choice.
Generalization, calibration, and honest evaluation are statistical claims governed by bias–variance, the CLT, and tests.
Generative models (VAEs, diffusion, autoregressive, GANs) are direct constructions from this book’s probability.
Probability and statistics are not background for ML — they are its foundation, with linear algebra and calculus as the computational tools.