Linear regression is linear in the parameters, not necessarily in the raw inputs. That loophole is enormous: by transforming and combining predictors we can model curves, interactions, categories, and saturating effects — all while keeping the simple machinery of least squares. This chapter is about feature engineering: building the columns of \(\mX\) so that a linear model can capture nonlinear reality.

Polynomial and basis-function regression

Replace \(x\) with a vector of basis functions \(\phi(x)=(1,x,x^2,\dots)\) and the model \(y=\vbeta^\top\phi(x)+\varepsilon\) is still linear in \(\vbeta\) — so OLS solves it unchanged. The fitted curve is nonlinear in \(x\) but the estimation is ordinary linear regression on the expanded design matrix. This “feature map” trick is the bridge from lines to arbitrary curves, and (Chapter [ch:nonparametric]) the seed of kernels and splines.

Categorical predictors: dummy coding

A categorical variable with \(k\) levels enters the model as \(k-1\) dummy (indicator) variables, each \(0/1\). One level is the reference; each dummy’s coefficient is the difference in mean response relative to the reference, holding other predictors fixed.

Including a dummy for all \(k\) levels and an intercept makes the columns sum to the intercept column — perfect collinearity, a singular \(\mX^\top\mX\). This is the dummy variable trap; always drop one level (or the intercept). Note that one-hot encoding in ML keeps all \(k\) columns precisely because regularization (or no explicit intercept) sidesteps the singularity.

Interactions

An interaction term \(x_1 x_2\) lets the effect of one predictor depend on another: in \(y=\beta_0+\beta_1x_1+\beta_2x_2+\beta_3 x_1x_2\), the slope of \(y\) in \(x_1\) is \(\beta_1+\beta_3 x_2\). Without the product term, the model forces effects to be purely additive.

“Does the effect of dose depend on age?” is an interaction question. A purely additive model answers “no” by assumption; adding \(x_1x_2\) lets the data decide. Deep networks discover such interactions automatically through their hidden layers — a feed-forward net is, in part, an interaction-and-transformation machine — which is one reason they need less manual feature engineering.

Transformations and scaling

  • Log transforms tame right-skew and turn multiplicative relationships additive; with a log response, coefficients read as approximate percentage changes (a log–log model gives elasticities).

  • Standardization (\(z=(x-\bar x)/s\)) puts predictors on a common scale so coefficients are comparable and — critically — so regularization (Chapter [ch:regularization]) and gradient descent behave well.

  • Box–Cox / Yeo–Johnson choose a power transform to stabilize variance and improve normality.

Data leakage via preprocessing is one of the most common and silent errors. Compute scaling parameters, imputation values, and transforms on the training fold only, then apply them to validation/test data. Fitting a scaler on the whole dataset before splitting leaks information about the test set into training and inflates your measured performance. Use a Pipeline so the transform is refit inside every cross-validation fold (Chapter [ch:biasvariance]).

Feature engineering is the original “representation learning.” Classical ML lived or died by hand-crafted features — polynomials, interactions, domain transforms — fed to a linear model. The deep-learning revolution automated this: hidden layers learn the feature map \(\phi(\vx)\) that earlier we built by hand, and the final linear layer is the regression on top. The kernel trick (Chapter [ch:nonparametric]) is the same idea taken to an implicit infinite-dimensional \(\phi\). Whether features are engineered or learned, the downstream model is still the linear regression of this book — which is why these fundamentals transfer directly.

import numpy as np
from sklearn.preprocessing import PolynomialFeatures, StandardScaler
from sklearn.linear_model import LinearRegression
from sklearn.pipeline import make_pipeline
rng = np.random.default_rng(0)
x = np.linspace(-3, 3, 80).reshape(-1, 1)
y = 0.5*x.ravel()**2 - x.ravel() + rng.normal(0, 0.4, 80)
# Pipeline keeps scaling + basis expansion leak-free inside CV
model = make_pipeline(StandardScaler(),
                      PolynomialFeatures(degree=2, include_bias=False),
                      LinearRegression()).fit(x, y)
print(model.score(x, y).round(3))   # R^2 of the quadratic-feature fit

Chapter summary

  • Linear regression is linear in parameters: basis functions \(\phi(x)\) let it fit curves while OLS stays unchanged.

  • Categorical predictors use \(k-1\) dummies; including all \(k\) with an intercept is the dummy variable trap.

  • Interactions (\(x_1x_2\)) let one predictor’s effect depend on another; networks learn these automatically.

  • Log transforms, standardization, and Box–Cox reshape features; scaling is essential for regularization and gradient descent.

  • Fit all preprocessing on training data only — preprocessing on the full set causes data leakage. Hidden layers automate feature engineering.