A regression can be technically correct and practically misleading. This chapter is the practitioner’s survival guide: how to measure performance with the right metric, the classic traps that fool even experts, and the bright line between prediction and causation — the distinction that determines whether a coefficient can guide a decision or merely describe an association.
Regression metrics
Common metrics for a continuous response:
MSE / RMSE: \(\tfrac1n\sum e_i^2\) and its square root; RMSE is in the response’s units and penalizes large errors heavily.
MAE: \(\tfrac1n\sum|e_i|\); robust to outliers, equal weight to all errors.
\(R^2\): fraction of variance explained (Chapter [ch:simple]); good for comparison, poor as an absolute quality measure.
MAPE: mean absolute percentage error; scale-free but undefined/unstable near zero.
The choice encodes your loss. If a $10k error is twice as bad as two $5k errors, use RMSE; if all errors hurt proportionally, use MAE. Reporting RMSE when the business cares about MAE (or vice versa) optimizes the wrong thing. Always compare against a baseline — predicting the mean, or last year’s value — so a number like “RMSE \(=4.2\)” has meaning.
Classic pitfalls
The hall of fame of regression mistakes:
Extrapolation: predictions outside the training range of \(\vx\) are unreliable — the fitted form may not hold there.
Data leakage: any information from the test set (or the future) entering training inflates measured performance and collapses in production — the #1 cause of “great in the notebook, terrible in production.”
Simpson’s paradox: a trend within every subgroup can reverse when groups are pooled; an omitted grouping variable flips the sign.
Multiple comparisons / \(p\)-hacking: test enough predictors and some look “significant” by chance.
Outliers and high-leverage points: a few extreme points can dominate the fit (Chapter [ch:robust]); always inspect them.
Overfitting to the validation set: tuning hundreds of configurations against one validation split silently overfits it — use nested CV.
Correlation is not causation
A regression coefficient measures association in the observed data, not the effect of intervening. Ice-cream sales predict drownings (both driven by summer heat — a confounder), but banning ice cream saves no one. To read a coefficient causally you need either a randomized experiment or strong, explicit assumptions (no unmeasured confounders) plus the right adjustment set. Adding controls can help (block confounders) or hurt (conditioning on a collider or mediator induces spurious associations) — which variables to include is a causal question a regression alone cannot answer.
A useful mantra: regression adjusts for what you put in it, and nothing else. It cannot adjust for variables you did not measure, and it will happily hand you a precise, significant, and completely non-causal coefficient. Modern causal inference (potential outcomes, DAGs, instrumental variables, difference-in-differences, regression discontinuity) builds on top of regression but adds the design and assumptions that license a causal reading.
A practical workflow
Plot the data and define the goal (predict vs. explain) and the loss/metric.
Split off a test set first; do all exploration and tuning on the rest.
Build features in a leak-free pipeline; start simple (linear) as a baseline.
Diagnose residuals; add complexity or regularization guided by cross-validation.
Report the right metric with uncertainty, against a baseline, on the untouched test set.
State assumptions explicitly — especially before any causal claim.
These lessons scale directly to large ML and AI systems. Data leakage (e.g. train/test contamination, or future information in features) is the most common reason a model that dazzles offline fails in deployment, and it is rampant in large pipelines. Distribution shift is extrapolation at scale — models degrade when production data drifts from training data. Spurious correlations / shortcut learning is Simpson’s paradox in disguise: deep nets latch onto confounded cues (backgrounds, watermarks) that do not generalize. And causal ML — uplift modeling, off-policy evaluation in RL, treatment-effect estimation — exists precisely because predictive accuracy does not imply you can act on a feature. Regression’s hard-won cautions are the foundation of trustworthy AI.
import numpy as np
from sklearn.metrics import mean_squared_error, mean_absolute_error, r2_score
rng = np.random.default_rng(0)
y = rng.normal(50, 10, 200)
yhat = y + rng.normal(0, 4, 200); yhat[:3] += 40 # a few big misses
print("RMSE:", round(mean_squared_error(y, yhat, squared=False), 2)) # outlier-sensitive
print("MAE :", round(mean_absolute_error(y, yhat), 2)) # outlier-robust
print("R^2 :", round(r2_score(y, yhat), 3))
# Always compare to a baseline (predict the mean):
print("baseline RMSE:", round(mean_squared_error(y, np.full_like(y, y.mean()),
squared=False), 2))Chapter summary
Pick the metric that matches your cost: RMSE (penalizes big errors), MAE (robust), \(R^2\) (relative); always compare to a baseline.
Beware extrapolation, data leakage, Simpson’s paradox, multiple comparisons, outliers, and validation-set overfitting.
A coefficient is association, not causation; causal reading needs experiments or explicit assumptions and the right adjustment set.
Regression adjusts only for variables you include — never for unmeasured confounders.
These pitfalls scale to ML as leakage, distribution shift, shortcut learning, and the motivation for causal ML.