Regression answers a deceptively simple question: given some inputs, what number should I predict for the output, and how are they related? Predict a house’s price from its size; a patient’s risk from their labs; tomorrow’s demand from today’s. This chapter sets up the vocabulary — response and predictors, the regression function, signal versus noise — and draws the line between prediction (guess the number well) and inference (understand the relationship), a distinction that shapes everything that follows.

The setup: response and predictors

In regression we model a response (or target) \(y\) as a function of one or more predictors (features, covariates) \(\vx=(x_1,\dots,x_p)\) plus random noise: \[y = f(\vx) + \varepsilon,\qquad \E[\varepsilon]=0.\] The goal is to estimate \(f\) from data \(\{(\vx_i,y_i)\}_{i=1}^n\). When \(y\) is continuous we call it regression; when \(y\) is a category, classification — but, as we will see, the same machinery handles both.

Think of the data as a cloud of points and \(f\) as the smooth trend running through it. The term \(\varepsilon\) is everything the predictors don’t explain — measurement error, omitted causes, genuine randomness. Regression tries to recover the trend while ignoring the scatter. The art is fitting the signal \(f\) without chasing the noise \(\varepsilon\), a tension we will formalize as the bias–variance tradeoff in Chapter [ch:biasvariance].

The regression function is the best prediction

What is the ideal \(f\)? If we measure error by squared difference, the answer is a conditional expectation.

The function that minimizes expected squared prediction error is the regression function — the conditional mean \[f(\vx)=\E[Y\mid \mathbf{X}=\vx].\] In words: the best guess for \(y\) at a given \(\vx\) is the average of all the \(y\)’s that occur there. Every regression method — linear models, trees, neural networks — is ultimately an attempt to approximate this conditional mean from a finite sample. Different methods just make different assumptions about its shape.

Prediction versus inference

Regression serves two distinct goals, and knowing which you are after changes how you model.

You only care that \(\hat y\) is accurate on new inputs — the model can be a black box. “Will this transaction be fraudulent?” Accuracy is king; interpretability is optional. This is the machine-learning mindset.

You want to understand the relationship: which predictors matter, in which direction, by how much, and with what certainty. “Does this drug lower blood pressure, controlling for age?” Here interpretable coefficients and honest uncertainty are essential. This is the classical-statistics mindset.

Confusing the two goals causes real damage. A model tuned purely for predictive accuracy may have coefficients that are uninterpretable or even sign-flipped by correlated predictors — so you cannot read causal meaning off them (Chapter [ch:diagnostics], [ch:pitfalls]). Conversely, a model built for clean inference may leave predictive accuracy on the table. Decide up front: are you trying to predict or to explain?

A first taxonomy of regression

The chapters ahead expand a single idea — approximate \(\E[Y\mid\vx]\) — along two axes: the form of \(f\) and the type of \(y\).

  • Linear (\(f(\vx)=\vbeta^\top\vx\)): simple and multiple linear regression (Chapters [ch:simple], [ch:multiple]); interpretable, fast, the default.

  • Generalized linear (a link function wraps a linear predictor): logistic for binary \(y\), Poisson for counts (Chapters [ch:logistic], [ch:glm]).

  • Nonlinear / nonparametric (flexible \(f\)): polynomials, splines, kernels, trees, Gaussian processes (Chapter [ch:nonparametric]) — and, at the extreme, neural networks (Chapter [ch:regmlai]).

Why regression matters

Regression is the most widely used statistical method in the world, and it is the conceptual seed of supervised machine learning. Its ideas — a parametric prediction function, a loss that measures error, optimization to fit parameters, regularization to generalize — recur at every scale, from a two-column spreadsheet to a billion-parameter network.

Supervised machine learning is regression and classification at scale. A neural network is a highly flexible regression function \(f_{\vtheta}(\vx)\) approximating \(\E[Y\mid\vx]\); its final layer is literally a linear (or logistic/softmax) regression on learned features; its training minimizes a regression loss (squared error) or classification loss (cross-entropy) by gradient descent. Linear regression is the “hello world” that introduces every concept you will reuse: features, weights, a loss, fitting, overfitting, and regularization. Understand regression deeply and the rest of ML is the same melody in a richer key.

import numpy as np
from sklearn.linear_model import LinearRegression
# Data following y = 0.78 x + 1.3 + noise
rng = np.random.default_rng(0)
x = np.linspace(1, 9, 40)
y = 0.78*x + 1.3 + rng.normal(0, 0.7, size=x.size)
model = LinearRegression().fit(x.reshape(-1, 1), y)
print(model.coef_[0].round(3), model.intercept_.round(3))   # ~0.78, ~1.3
print(model.predict([[5.0]]).round(2))                       # predict y at x=5

Chapter summary

  • Regression models a response as \(y=f(\vx)+\varepsilon\) and estimates the trend \(f\) from data while ignoring noise.

  • The ideal predictor under squared error is the regression function \(f(\vx)=\E[Y\mid\vx]\) — the conditional mean.

  • Distinguish prediction (accurate \(\hat y\)) from inference (understanding the relationship); the goal shapes the model.

  • Regression spans linear, generalized linear, and nonparametric forms — all approximating \(\E[Y\mid\vx]\).

  • It is the conceptual core of supervised ML: a neural net is flexible regression with a regression/classification output layer.