A fitted model always produces coefficients — but whether you can trust them, predict with them, or attach uncertainty to them depends on assumptions. This chapter states what OLS assumes, shows how to check it with residual plots, and develops the inference machinery (standard errors, \(t\)- and \(F\)-tests, confidence and prediction intervals) that turns point estimates into honest conclusions.

The classical assumptions

The standard linear-regression assumptions, often abbreviated LINE, are:

  1. Linearity: \(\E[y\mid\vx]=\vbeta^\top\vx\) — the mean is linear in the predictors.

  2. Independence: the errors \(\varepsilon_i\) are independent across observations.

  3. Normality: \(\varepsilon_i\) are (approximately) Gaussian — needed for exact small-sample inference.

  4. Equal variance (homoscedasticity): \(\Var(\varepsilon_i)=\sigma^2\) is constant.

The Gauss–Markov theorem says that under linearity, independence, zero-mean errors, and constant variance (normality not required), OLS is BLUE — the Best Linear Unbiased Estimator, meaning it has the smallest variance among all unbiased estimators that are linear in \(\vy\). This is why least squares is the default: it is optimal in a precise sense, for free, without distributional assumptions. Normality is needed only for exact \(t\)/\(F\) inference; for large \(n\) the Central Limit Theorem rescues us.

Diagnostics: read the residuals

The residuals are your window into assumption violations. A handful of plots catches most problems.

Should be a structureless cloud around zero. A curve signals missing nonlinearity (transform or add terms); a funnel signals heteroscedasticity.

Residual quantiles vs. normal quantiles; points off the diagonal in the tails reveal non-normality or outliers.

Detect non-constant spread and high-leverage / high-influence points (Cook’s distance).

Inference: standard errors and tests

Under the assumptions, the coefficient estimator is Gaussian with covariance \[\widehat{\Cov}(\bhat)=\hat\sigma^2(\mX^\top\mX)^{-1},\qquad \hat\sigma^2=\frac{\RSS}{n-p-1}.\] The square roots of the diagonal are the standard errors \(\mathrm{SE}(\hat\beta_j)\).

These give the standard tests:

  • \(t\)-test for a single coefficient: \(t_j=\hat\beta_j/\mathrm{SE}(\hat\beta_j)\) tests \(H_0:\beta_j=0\). The \(p\)-value answers “could this effect plausibly be zero?”

  • \(F\)-test for the whole model (or a group of coefficients): tests whether the predictors jointly explain variance beyond the intercept.

  • Confidence interval: \(\hat\beta_j\pm t^\ast\,\mathrm{SE}(\hat\beta_j)\) — plausible values for the true effect.

A coefficient is “significant” when it is large relative to its uncertainty. Doubling the sample size shrinks standard errors (roughly as \(1/\sqrt n\)), so with enough data even tiny effects become statistically significant — which is why effect size matters more than the \(p\)-value alone.

Confidence vs. prediction intervals

Do not confuse a confidence interval for the mean response with a prediction interval for a new observation. The confidence interval captures uncertainty about the average \(y\) at a given \(\vx\) (only estimation error). The prediction interval also includes the irreducible noise \(\sigma^2\) of a single draw, so it is always wider. Reporting a narrow confidence interval when a user actually needs a prediction interval drastically understates real-world uncertainty.

Multicollinearity and the VIF

The variance inflation factor for predictor \(j\) is \(\mathrm{VIF}_j=1/(1-R_j^2)\), where \(R_j^2\) is from regressing \(x_j\) on the other predictors. It measures how much collinearity inflates \(\Var(\hat\beta_j)\). Rules of thumb: \(\mathrm{VIF}>5\) is concerning, \(>10\) is severe.

The assumptions explain when a model’s coefficients are trustworthy — directly relevant to interpretable / explainable ML and to causal inference. Heteroscedasticity motivates weighted least squares and, in deep learning, heteroscedastic loss heads that predict both a mean and a variance (aleatoric uncertainty). Multicollinearity is why correlated features destabilize linear models and why regularization (Chapter [ch:regularization]) and decorrelating transforms like PCA are everyday tools. And the covariance \(\hat\sigma^2(\mX^\top\mX)^{-1}\) is the classical ancestor of the Laplace approximation used for uncertainty in Bayesian neural networks.

import numpy as np, statsmodels.api as sm
rng = np.random.default_rng(1)
n = 100
x1 = rng.normal(size=n); x2 = rng.normal(size=n)
y = 1 + 2*x1 - 0.5*x2 + rng.normal(0, 1, n)
X = sm.add_constant(np.column_stack([x1, x2]))
fit = sm.OLS(y, X).fit()
print(fit.summary())            # coefficients, SEs, t-stats, p-values, R^2, F-stat
print(fit.conf_int())           # 95% confidence intervals for each beta

Chapter summary

  • OLS assumes LINE: linearity, independence, normality (for exact inference), equal variance.

  • Gauss–Markov: OLS is BLUE under the mean/variance assumptions — normality is needed only for small-sample \(t\)/\(F\) tests.

  • Diagnose with residual plots: curves \(\to\) nonlinearity, funnels \(\to\) heteroscedasticity, Q–Q tails \(\to\) non-normality/outliers.

  • Standard errors from \(\hat\sigma^2(\mX^\top\mX)^{-1}\) drive \(t\)-tests, \(F\)-tests, and confidence intervals; prediction intervals are wider than confidence intervals.

  • The VIF flags multicollinearity, which destabilizes coefficients and motivates regularization.