This capstone ties the book together by arguing a single thesis: regression is not just one tool in machine learning — it is the conceptual skeleton of the entire field. Supervised learning is regression and classification at scale. A neural network is a flexible regression function. Its output layer is a linear, logistic, or softmax regression. Its loss is a regression or classification likelihood. Its training is least-squares-style optimization. Even reinforcement learning and generative AI reduce, at their core, to regression problems. We revisit each idea from the book and show where it lives in modern AI.
The neural network as flexible regression
A feed-forward neural network computes \[f_{\vtheta}(\vx)=\mathbf W^{(L)}\,\phi\!\big(\cdots\phi(\mathbf W^{(1)}\vx+\vb^{(1)})\cdots\big)+\vb^{(L)},\] which is exactly the basis-function regression of Chapter [ch:features] — except the basis \(\phi(\cdot)\) is learned rather than hand-designed. Each layer is the matrix multiply \(\mX\mathbf W\) of Chapter [ch:multiple]; the final layer is a multiple/logistic/softmax regression on the learned representation. The universal approximation theorem guarantees this flexible regressor can approximate any continuous function. Deep learning is feature learning plus regression.
Loss functions are negative log-likelihoods
The loss a network minimizes is precisely the GLM correspondence of Chapter [ch:glm]: the output activation and loss together specify the response’s distribution, and the loss is its negative log-likelihood.
| Task | Output activation | Loss | Regression analog |
|---|---|---|---|
| Continuous value | identity (linear) | MSE / \(L_2\) | linear regression (Ch. [ch:simple]) |
| Robust regression | identity | Huber / smooth-\(L_1\) | robust regression (Ch. [ch:robust]) |
| Binary label | sigmoid | binary cross-entropy | logistic regression (Ch. [ch:logistic]) |
| Multiclass label | softmax | categorical cross-entropy | softmax regression (Ch. [ch:logistic]) |
| Counts | exp (log link) | Poisson loss | Poisson GLM (Ch. [ch:glm]) |
| Quantiles / intervals | identity | pinball loss | quantile regression (Ch. [ch:robust]) |
Choosing a network’s head and loss is choosing a GLM. The right loss is never arbitrary — it is the likelihood of the data.
Optimization: least squares by gradient descent
Linear and logistic regression have closed forms or quickly-converging solvers; deep networks do not, so they minimize the same losses by gradient descent and its stochastic variants (SGD, Adam, AdamW). The objective is identical in spirit to minimizing \(\RSS\) — only the parameter count and nonconvexity change. Backpropagation is just the chain rule computing the gradient of the regression loss with respect to every weight. The intuition you built fitting a line carries all the way to training billion-parameter models.
Regularization, revisited at scale
Weight decay \(=\) ridge (\(L_2\)) penalty (Chapter [ch:regularization]); the default in AdamW for training Transformers.
\(L_1\) / sparsity penalties prune and compress networks.
Early stopping and dropout are implicit regularizers managing the same bias–variance tradeoff (Chapter [ch:biasvariance]).
Cross-validation remains the universal protocol for tuning every hyperparameter.
Regression at the frontier of AI
The regression view illuminates even the most modern systems:
Large language models predict the next token by softmax regression over the vocabulary on top of a Transformer’s learned representation — cross-entropy, exactly as in Chapter [ch:logistic]. Adapting an LLM to output numbers uses lightweight regression heads (a linear layer, or iterative-refinement heads like RELISH) trained with MSE.
Diffusion models generate images by training a network to regress the noise added to data (an MSE objective) — generation as denoising regression.
Reinforcement learning learns value functions by regressing returns (TD/Monte-Carlo targets); distributional RL uses quantile regression (Chapter [ch:robust]) to predict the whole return distribution.
Scaling laws are themselves power-law regressions of loss against model size, data, and compute — regression used to forecast the behavior of AI systems.
Uncertainty & safety: Bayesian regression (Chapter [ch:robust]) seeds Bayesian neural networks, MC-dropout, and Laplace approximations so models know what they don’t know.
The thread, end to end
Trace one idea through the whole book and into AI: fit a function to data by minimizing a loss, and regularize so it generalizes. Chapter [ch:simple] fit a line by least squares; Chapter [ch:multiple] made it multivariate and geometric; Chapter [ch:logistic] swapped the loss for cross-entropy; Chapter [ch:glm] unified losses as likelihoods; Chapter [ch:regularization] added penalties; Chapter [ch:biasvariance] explained generalization; Chapter [ch:nonparametric] made the function flexible. A deep network is all of these at once, learned end to end. Regression is where machine learning begins, and it never really leaves.
import torch, torch.nn as nn
# A 1-layer net with identity output + MSE == linear regression by gradient descent
X = torch.randn(200, 3); beta = torch.tensor([2.0, -1.0, 0.5])
y = X @ beta + 0.1*torch.randn(200)
net = nn.Linear(3, 1)
opt = torch.optim.AdamW(net.parameters(), lr=0.05, weight_decay=1e-3) # = ridge
lossf = nn.MSELoss()
for _ in range(400):
opt.zero_grad()
loss = lossf(net(X).squeeze(), y) # minimize RSS by gradient descent
loss.backward(); opt.step()
print(net.weight.data.round(decimals=3)) # recovers ~[2.0, -1.0, 0.5]Chapter summary
A neural network is flexible regression: learned features \(\phi(\vx)\) plus a linear/logistic/softmax head (Chapters [ch:multiple], [ch:logistic]).
Its loss is a negative log-likelihood — output activation \(+\) loss \(=\) a GLM (Chapter [ch:glm]); the right loss is the data’s likelihood.
Training minimizes that loss by gradient descent (backprop \(=\) chain rule), generalizing least squares.
Weight decay is ridge; dropout, early stopping, and cross-validation manage the bias–variance tradeoff at scale.
LLMs (softmax/regression heads), diffusion (noise regression), RL (value/quantile regression), and scaling laws are all regression — the idea that starts the book and underpins modern AI.