A model that fits the training data perfectly is often useless on new data. This chapter explains why — the bias–variance tradeoff, the most important concept in all of predictive modeling — and gives the tools to measure generalization honestly: train/test splits, cross-validation, and information criteria. These ideas govern every choice from polynomial degree to neural-network size.

Overfitting and underfitting

A model underfits when it is too simple to capture the true pattern (high error on both training and new data). It overfits when it is so flexible that it memorizes noise in the training data (low training error, high error on new data). The goal is the sweet spot between them.

Decomposing the error

For squared-error loss, the expected test error at a point decomposes exactly into three pieces: \[\E\big[(y-\hat f(\vx))^2\big]=\underbrace{\big(\text{Bias}[\hat f(\vx)]\big)^2}_{\text{too rigid}}+\underbrace{\Var[\hat f(\vx)]}_{\text{too sensitive}}+\underbrace{\sigma^2}_{\text{irreducible}}.\] Bias is error from wrong assumptions (a line can’t fit a curve). Variance is sensitivity to the particular training sample (a wiggly curve changes wildly with new data). Irreducible noise \(\sigma^2\) is the floor no model can beat. Simplifying lowers variance but raises bias; flexibility does the reverse — you cannot drive both to zero, so you balance them.

Picture hitting a target with darts. High bias, low variance: tight cluster, but off-center (consistent and wrong). Low bias, high variance: centered on average, but scattered everywhere (right on average, wrong each time). Good prediction needs both centered and tight — and since data is finite, getting there means deliberately accepting a little bias (via regularization) to buy a large reduction in variance.

Honest evaluation: train/test and cross-validation

Estimate generalization on data the model has not seen.

  • Train/validation/test split: fit on train, tune on validation, report once on test.

  • \(k\)-fold cross-validation: split data into \(k\) folds; train on \(k-1\), validate on the held-out fold, rotate, and average. Uses all data for both roles and gives a lower-variance performance estimate.

  • Leave-one-out (LOOCV): the extreme \(k=n\); for linear models it has a remarkable closed form via the hat-matrix diagonal (no refitting needed).

Information criteria

For models fit by maximum likelihood, AIC and BIC estimate out-of-sample quality without a held-out set by penalizing complexity: \[\mathrm{AIC}=-2\ln\hat L+2k,\qquad \mathrm{BIC}=-2\ln\hat L+k\ln n,\] where \(k\) is the number of parameters. Lower is better; BIC penalizes complexity more harshly (and favors smaller models) as \(n\) grows. They are fast surrogates for cross-validation when refitting is expensive.

The cardinal sin of evaluation is letting the test set influence the model. If you tune hyperparameters on the test set, peek at it repeatedly, or preprocess before splitting, your reported error is optimistic and your model will disappoint in production. Lock the test set away; do all selection with cross-validation on the training data. Also beware that with non-IID data (time series, grouped/clustered data) ordinary \(k\)-fold leaks — use time-series or grouped splits instead.

The bias–variance tradeoff is the organizing principle of all of machine learning: model capacity, regularization strength, tree depth, network width, dropout, and early stopping are all knobs on this single dial. Cross-validation is the universal protocol for hyperparameter tuning. Intriguingly, very large neural networks exhibit double descent: past the interpolation threshold, test error can fall again even as the classical curve says it should rise — a refinement, not a refutation, of this framework, and an active research frontier. The intuition you build here remains the foundation for reasoning about every model’s generalization.

import numpy as np
from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import LinearRegression
from sklearn.pipeline import make_pipeline
from sklearn.model_selection import cross_val_score
rng = np.random.default_rng(0)
x = np.sort(rng.uniform(-3, 3, 60)).reshape(-1, 1)
y = np.sin(x).ravel() + rng.normal(0, 0.2, 60)
for d in [1, 3, 9, 15]:
    model = make_pipeline(PolynomialFeatures(d), LinearRegression())
    mse = -cross_val_score(model, x, y, cv=5,
                           scoring="neg_mean_squared_error").mean()
    print(f"degree {d:2d}: CV MSE = {mse:.3f}")   # U-shaped: best at moderate degree

Chapter summary

  • Underfitting = too rigid (high bias); overfitting = memorizing noise (high variance). Aim for the middle.

  • Expected squared error \(=\) bias\(^2+\) variance \(+\) irreducible noise \(\sigma^2\); you trade bias against variance, never eliminate both.

  • Measure generalization on unseen data: train/test splits and \(k\)-fold cross-validation; LOOCV has a closed form for linear models.

  • AIC/BIC approximate out-of-sample quality by penalizing parameters; BIC penalizes harder.

  • Never let the test set leak into modeling; this tradeoff governs every ML capacity knob (with double descent as a modern twist).