Preface: how to use this book
Regression is the oldest and most useful idea in data analysis: predict a number from other numbers, and understand the relationship between them. Two centuries after Gauss and Legendre fit lines to astronomical data, regression is still the workhorse of statistics — and, in a deep sense, the conceptual core of machine learning. A neural network is, at heart, an enormously flexible regression function; the loss it minimizes is a regression or classification loss; the output layer that turns features into a prediction is a regression. This book builds regression from a single line through a cloud of points all the way to the methods behind modern AI.
Who this is for
This book starts from the very beginning — what it means to fit a line — and climbs to regularization, generalized linear models, splines and kernels, robust and Bayesian regression, and the regression view of deep learning. It is for the self-learner who wants both fluency (“I can fit the model, read the output, and trust it”) and understanding (“I know what the coefficients mean, when the model breaks, and why these methods generalize”). Every major idea is motivated with a picture, a worked example, and a forward link to machine learning.
The central question
All of regression orbits one question and the ways we answer it:
The prediction problem: given inputs \(\vx\), predict an output \(y\). The best prediction (in squared-error terms) is the regression function \(\E[Y\mid \vx]\).
The estimation problem: we never know \(\E[Y\mid\vx]\); we estimate it from data by choosing a model and fitting its parameters — usually by least squares or maximum likelihood.
The generalization problem: a model must predict well on new data, not just fit the sample — the tension that drives regularization and validation.
The structure
Part I — Foundations (beginner): what regression is; simple linear regression.
Part II — The Linear Model (core): multiple regression and its geometry; assumptions, diagnostics, and inference; feature engineering.
Part III — Generalization (core): the bias–variance tradeoff and validation; regularization (ridge, lasso, elastic net).
Part IV — Beyond the Linear Model (advanced): logistic regression; generalized linear models; nonlinear and nonparametric regression.
Part V — Advanced & Applications: robust, quantile, and Bayesian regression; evaluation, pitfalls, and causality; and a capstone on regression in ML/DL/AI.
How to read it
Read with a pen and a computer. When you meet a Definition, restate it; when you meet an Example, try it first; when you meet a Try it in code box, run it. The colored boxes recur throughout:
Key idea — the one sentence to remember.
Intuition — the geometric or statistical picture.
Common pitfall — the mistakes that bite everyone.
How this powers ML / AI — the forward link to applications.
Try it in code — a hands-on check in Python (NumPy / scikit-learn).
Hold one picture above all others: regression draws the best curve through a cloud of data, where “best” means smallest error. Everything else — multiple predictors, regularization, logistic and generalized models, splines, neural networks — is a variation on which curves are allowed and how we measure error. Master the line, and the rest of the subject, all the way up to deep learning, is the same idea made flexible.