We now cross from probability into statistics — from “given the model, predict the data” to “given the data, infer the model.” The central task is estimation: using a sample to guess an unknown parameter. The dominant method, maximum likelihood, is not just a statistical technique; it is the principle from which nearly every loss function in machine learning is derived. This is one of the most important chapters in the book.

Estimators and their quality

An estimator \(\hat\theta\) is a function of the data used to estimate an unknown parameter \(\theta\). Because it depends on a random sample, \(\hat\theta\) is itself a random variable with a sampling distribution.

We judge estimators by three properties:

  • Bias: \(\Bias(\hat\theta)=\E[\hat\theta]-\theta\). Unbiased means correct on average.

  • Variance: \(\Var(\hat\theta)\) — how much the estimate jitters across samples.

  • Mean squared error: \(\MSE(\hat\theta)=\E[(\hat\theta-\theta)^2]=\Bias^2+\Var\).

The decomposition \(\MSE=\Bias^2+\Var\) is the seed of the bias–variance tradeoff (Chapter [ch:regression]) that governs all of machine learning. A more flexible estimator/model lowers bias but raises variance; a simpler one does the reverse. The best estimator is not always unbiased — accepting a little bias to cut variance often lowers total error, which is exactly what regularization does.

The sample variance with divisor \(n\) is biased low; dividing by \(n-1\) instead (Bessel’s correction) makes it unbiased. The maximum-likelihood estimate of a Gaussian’s variance uses \(n\) and is therefore slightly biased — a concrete reminder that MLE is not automatically unbiased.

The likelihood function

Given data \(\Ldata=\{x_1,\dots,x_n\}\) and a model with parameter \(\theta\), the likelihood is the probability of the observed data viewed as a function of \(\theta\): \[L(\theta)=P(\Ldata\given\theta)=\prod_{i=1}^n p(x_i\given\theta)\quad(\text{i.i.d.\ data}).\]

Likelihood is not a probability distribution over \(\theta\) — it does not integrate to \(1\) over \(\theta\), and “\(L(\theta)\)” answers “how well does this \(\theta\) explain the data I saw,” not “how probable is \(\theta\).” Treating likelihood as a posterior is the error that Bayes’ theorem (and a prior) exists to correct. The probability is over the data; we merely read it as a function of \(\theta\).

Maximum likelihood estimation

The maximum likelihood estimator is the parameter that makes the observed data most probable: \[\hat\theta_{\text{MLE}}=\argmax_\theta L(\theta)=\argmax_\theta \sum_{i=1}^n \log p(x_i\given\theta).\] We maximize the log-likelihood because the log turns the product into a sum (numerically stable, easy to differentiate) without moving the maximum.

MLE asks a beautifully simple question: “of all the settings of \(\theta\), which one would have made my data least surprising?” Pick that one. Geometrically, you slide \(\theta\) until the model’s probability mass piles up where your data actually fell. Maximizing is done with the calculus of the companion volume: set the derivative of the log-likelihood to zero, or descend its negative by gradient descent.

Observe \(k\) heads in \(n\) flips. The log-likelihood is \(k\log p+(n-k)\log(1-p)\); setting its derivative to zero gives the unsurprising \(\hat p=k/n\), the observed frequency. MLE recovers common sense — and then generalizes it to models far too complex for intuition.

Under regularity conditions the MLE is consistent (converges to the truth as \(n\to\infty\)), asymptotically normal, and asymptotically efficient (attains the Cramér–Rao lower bound — the smallest possible variance). The Fisher information quantifies how sharply peaked the likelihood is and sets that minimum variance.

Maximum likelihood is where loss functions come from. Minimizing the negative log-likelihood (NLL) is maximizing likelihood, and the choice of probability model determines the loss:

  • Categorical labels \(\Rightarrow\) NLL \(=\) cross-entropy loss (every classifier, every language model).

  • Gaussian targets \(\Rightarrow\) NLL \(=\) mean-squared error (regression).

  • Poisson counts \(\Rightarrow\) Poisson regression loss; and so on.

So training a neural network by minimizing cross-entropy is maximum likelihood estimation of its parameters. The bias–variance decomposition explains why pure MLE overfits on small data, motivating the regularizers (= priors, Chapter [ch:bayesian]) that every real system adds. The MLE skeleton is underneath essentially all of supervised deep learning.

import numpy as np
from scipy.optimize import minimize_scalar
# MLE for a coin: maximize log-likelihood of k heads in n flips
k, n = 7, 10
nll = lambda p: -(k*np.log(p) + (n-k)*np.log(1-p))   # negative log-likelihood
res = minimize_scalar(nll, bounds=(1e-6, 1-1e-6), method='bounded')
print(round(res.x, 3))                  # 0.7 = k/n, the observed frequency

# MLE for a Gaussian recovers the sample mean and (biased) variance
x = np.random.normal(5, 2, size=10_000)
print(round(x.mean(),3), round(x.var(),3))   # mu_hat ~5, sigma^2_hat ~4 (divisor n)

Chapter summary

  • An estimator is a random function of data; judge it by bias, variance, and \(\MSE=\Bias^2+\Var\).

  • The likelihood \(L(\theta)=P(\Ldata\given\theta)\) is the data’s probability read as a function of \(\theta\) — not a distribution over \(\theta\).

  • MLE picks \(\theta\) maximizing the (log-)likelihood; it is consistent, asymptotically normal, and efficient, but can be biased.

  • Minimizing negative log-likelihood yields cross-entropy (categorical) and MSE (Gaussian) — the standard ML losses.

  • The bias–variance decomposition foreshadows why regularization (priors) is needed beyond pure MLE.