Regression is where probability and statistics become predictive machine learning. A statistical model describes how outputs depend on inputs plus randomness; regression fits that dependence to data. Linear and logistic regression are the simplest such models — and, not coincidentally, the conceptual templates for the entire supervised-learning toolbox, including deep networks. This chapter also formalizes the bias–variance tradeoff that governs every fitting decision you will ever make.

The linear model

Linear regression models a continuous target as a linear function of features plus Gaussian noise: \[y = \vw^\top\vx + b + \varepsilon,\qquad \varepsilon\sim\Normal(0,\sigma^2).\]

Fitting this model by maximum likelihood is identical to minimizing least squares. Because the noise is Gaussian, the log-likelihood of the data is \(-\frac{1}{2\sigma^2}\sum_i(y_i-\vw^\top\vx_i)^2\) plus constants — so maximizing likelihood is minimizing the sum of squared errors. “Least squares,” invented by Gauss and Legendre, turns out to be MLE under a Gaussian noise assumption. The probability model and the loss function are two views of one thing.

Minimizing squared error has the famous closed form (the normal equations of the linear-algebra companion), \(\hat{\vw}=(\mathbf{X}^\top\mathbf{X})^{-1}\mathbf{X}^\top\vy\) — statistics and linear algebra meeting exactly.

Logistic regression

For classification, we model the probability of a class rather than a continuous value.

Logistic regression passes a linear score through the sigmoid to produce a probability: \[P(y=1\given\vx)=\sigma(\vw^\top\vx+b),\qquad \sigma(z)=\frac{1}{1+e^{-z}}.\]

Fitting it by maximum likelihood means minimizing cross-entropy (the Bernoulli NLL). There is no closed form, so we use gradient descent — and the gradient has the clean “predicted minus actual” form from the calculus companion. The multi-class version uses the softmax and is exactly the final layer of a neural-network classifier.

Logistic regression is a one-layer neural network with a sigmoid output. Stack more layers with nonlinearities and you get deep learning; the loss (cross-entropy), the fitting principle (maximum likelihood), and the optimizer (gradient descent) are unchanged. This is why logistic regression is called the conceptual ancestor of modern classifiers — understand it probabilistically and you understand the output layer of essentially every deep model.

Generalized linear models

Linear and logistic regression are special cases of generalized linear models (GLMs): pick a distribution for the target (Gaussian, Bernoulli, Poisson, …) and a link function connecting its mean to a linear predictor \(\vw^\top\vx\). The distribution determines the loss (via MLE), unifying a whole family of models under one probabilistic principle — and explaining why so many losses look related.

The bias–variance tradeoff

For squared-error loss, the expected test error of a model decomposes as \[\E\big[(y-\hat f(\vx))^2\big] = \underbrace{\Bias^2}_{\text{too rigid}} + \underbrace{\Var}_{\text{too sensitive}} + \underbrace{\sigma^2}_{\text{irreducible noise}}.\]

This is the central tension of model fitting. A model that is too simple (high bias) underfits — it cannot capture the pattern. A model that is too flexible (high variance) overfits — it memorizes the training noise and varies wildly across samples. Generalization error is minimized in between. Regularization (= a prior, Chapter [ch:bayesian]) deliberately adds a little bias to remove a lot of variance; cross-validation estimates the test error so we can tune the balance.

This chapter is the skeleton of supervised learning. Linear regression \(=\) MLE under Gaussian noise \(=\) least squares; logistic/softmax regression \(=\) MLE under Bernoulli/Categorical \(=\) cross-entropy — and the latter is exactly the output layer of a deep classifier. GLMs explain the menagerie of losses as one principle. The bias–variance tradeoff is the lens for every capacity decision: network size, depth, regularization strength, early stopping, data augmentation, and ensembling all move you along this curve. (Deep learning adds a famous twist — the “double descent” where massively overparameterized models generalize well anyway — but the decomposition remains the foundational vocabulary.)

import numpy as np
# Linear regression by least squares == Gaussian-noise MLE (normal equations)
X = np.column_stack([np.ones(200), np.random.randn(200, 2)])
w_true = np.array([1.0, 2.0, -0.5])
y = X @ w_true + np.random.normal(0, 0.3, 200)
w_hat = np.linalg.lstsq(X, y, rcond=None)[0]
print(w_hat.round(2))                         # ~ [1, 2, -0.5]

# Ridge = MAP with a Gaussian prior: add lambda*I before inverting
lam = 1.0
w_ridge = np.linalg.solve(X.T@X + lam*np.eye(3), X.T@y)
print(w_ridge.round(2))                       # shrunk slightly toward 0

Chapter summary

  • Linear regression \(=\) MLE under Gaussian noise \(=\) least squares (closed-form normal equations).

  • Logistic/softmax regression \(=\) MLE under Bernoulli/Categorical \(=\) cross-entropy \(=\) a neural net’s output layer.

  • GLMs unify these: choose a distribution + link, and MLE fixes the loss.

  • The bias–variance tradeoff (\(\MSE=\Bias^2+\Var+\sigma^2\)) governs over/underfitting; regularization trades bias for variance.

  • Cross-validation estimates test error to tune the balance; these ideas drive every model-capacity decision in ML.