We begin where regression itself began: fitting a straight line through points. Simple linear regression — one predictor, one response — is the entire subject in miniature. Every concept you will need later (a model, a loss, least-squares fitting, goodness of fit, interpretation) appears here in its clearest form. Master the line and the rest is generalization.

The model

Simple linear regression models the response as a straight-line function of one predictor plus noise: \[y_i = \beta_0 + \beta_1 x_i + \varepsilon_i,\] where \(\beta_0\) is the intercept, \(\beta_1\) the slope, and \(\varepsilon_i\) the error for observation \(i\).

The slope \(\beta_1\) is the engine of interpretation: it is the expected change in \(y\) for a one-unit increase in \(x\). The intercept \(\beta_0\) is the predicted \(y\) when \(x=0\) (often an extrapolation, sometimes meaningless). Fitting the model means choosing the line — the pair \((\beta_0,\beta_1)\) — that best matches the data. “Best” needs a definition, and that definition is least squares.

Least squares

The residual for point \(i\) is \(e_i=y_i-\hat y_i\), the vertical gap between the data and the line. Ordinary least squares (OLS) chooses \(\hat\beta_0,\hat\beta_1\) to minimize the residual sum of squares \[\RSS=\sum_{i=1}^n e_i^2 = \sum_{i=1}^n\big(y_i-\beta_0-\beta_1 x_i\big)^2.\]

Why squared errors? Squaring makes all errors positive, penalizes large misses disproportionately, yields a smooth function with a unique minimum found by calculus, and — as Chapter [ch:diagnostics] shows — corresponds to assuming Gaussian noise (so least squares is maximum likelihood). Setting the derivatives of \(\RSS\) to zero gives clean closed-form estimates: \[\hat\beta_1=\frac{\sum_i (x_i-\bar x)(y_i-\bar y)}{\sum_i (x_i-\bar x)^2}=\frac{\Cov(x,y)}{\Var(x)},\qquad \hat\beta_0=\bar y-\hat\beta_1\bar x.\]

Two facts worth remembering: the slope is the covariance of \(x\) and \(y\) divided by the variance of \(x\), and the fitted line always passes through the mean point \((\bar x,\bar y)\).

How good is the fit? \(R^2\)

The coefficient of determination measures the fraction of the response’s variance explained by the model: \[R^2 = 1-\frac{\RSS}{\TSS},\qquad \TSS=\sum_i (y_i-\bar y)^2,\] where \(\TSS\) is the total variance around the mean. \(R^2\) ranges from \(0\) (no better than predicting \(\bar y\)) to \(1\) (perfect fit).

\(R^2\) compares your line to the dumbest sensible model — the flat line at \(\bar y\). It answers “how much of the wiggle in \(y\) did my predictor account for?” In simple regression, \(R^2\) equals the squared correlation between \(x\) and \(y\). It is intuitive but easily abused, as the pitfall warns.

\(R^2\) has sharp edges. (1) It never decreases when you add predictors, so a high \(R^2\) can simply mean an overstuffed model — use adjusted \(R^2\) or validation instead (Chapters [ch:diagnostics], [ch:biasvariance]). (2) A high \(R^2\) does not mean the model is correct, the relationship is linear, or that \(x\) causes \(y\). (3) Anscombe’s quartet famously shows four wildly different datasets with identical \(R^2\) and identical fitted lines — so always plot the data, never trust a single number.

Interpreting and using the line

A fitted simple regression supports three uses: describe the association (sign and size of the slope), predict \(\hat y\) at a new \(x\), and — with the inferential tools of Chapter [ch:diagnostics] — test whether the slope is reliably nonzero. Keep predictions within the range of the observed \(x\); extrapolating a line far beyond the data is one of regression’s most common and dangerous mistakes.

Simple linear regression is the atom from which deep learning is built. A single artificial neuron computes exactly \(\hat y=\vw^\top\vx+b\) — a linear regression — optionally followed by a nonlinearity. The squared-error loss minimized here is the MSE loss used to train regression networks; the closed-form normal equations generalize to the matrix form of Chapter [ch:multiple]; and when no closed form exists (as in deep nets) we minimize the same \(\RSS\) by gradient descent. Understanding how slope, intercept, residuals, and \(R^2\) behave gives you correct intuition for the weights, biases, errors, and fit quality of any model.

import numpy as np
x = np.array([1,2,3,4,5,6,7,8,9], float)
y = np.array([2.3,3.4,3.2,4.0,4.8,5.5,6.4,7.2,8.0])
# Closed-form least squares
b1 = np.cov(x, y, bias=True)[0,1] / np.var(x)
b0 = y.mean() - b1*x.mean()
yhat = b0 + b1*x
R2 = 1 - ((y-yhat)**2).sum() / ((y-y.mean())**2).sum()
print(round(b0,3), round(b1,3), round(R2,4))     # intercept, slope, R^2
print(round(np.corrcoef(x,y)[0,1]**2, 4))         # equals R^2 in simple regression

Chapter summary

  • Simple linear regression fits \(y=\beta_0+\beta_1 x+\varepsilon\); the slope is the change in \(y\) per unit \(x\).

  • Least squares minimizes \(\RSS=\sum e_i^2\), giving closed forms \(\hat\beta_1=\Cov(x,y)/\Var(x)\), \(\hat\beta_0=\bar y-\hat\beta_1\bar x\).

  • The line passes through \((\bar x,\bar y)\); squared error corresponds to a Gaussian-noise / maximum-likelihood assumption.

  • \(R^2=1-\RSS/\TSS\) is the fraction of variance explained — intuitive but easily abused; always plot the data (Anscombe).

  • A neuron is a linear regression; this chapter’s loss and fitting recur throughout ML.