When predictors are many or correlated, ordinary least squares overfits: coefficients explode, variance soars, and predictions on new data suffer. Regularization fixes this by adding a penalty that discourages large coefficients — deliberately introducing a little bias to slash variance. Ridge, lasso, and elastic net are the three canonical penalties, and weight decay — the default in deep learning — is exactly ridge.

The idea: penalize complexity

Instead of minimizing fit alone, minimize fit plus a penalty on coefficient size: \[\min_{\vbeta}\ \underbrace{\norm{\vy-\mX\vbeta}^2}_{\text{fit}}+\lambda\underbrace{\,\Omega(\vbeta)}_{\text{penalty}} .\] The tuning parameter \(\lambda\ge 0\) sets the strength: \(\lambda=0\) recovers OLS; \(\lambda\to\infty\) shrinks all coefficients to zero. Choosing \(\lambda\) is navigating the bias–variance tradeoff of Chapter [ch:biasvariance], and it is chosen by cross-validation.

Ridge (\(L_2\))

Ridge regression penalizes the squared \(L_2\) norm, \(\Omega(\vbeta)=\sum_j\beta_j^2\), and has a closed form: \[\bhat_{\text{ridge}}=(\mX^\top\mX+\lambda\mathbf I)^{-1}\mX^\top\vy .\]

The added \(\lambda\mathbf I\) literally lifts the diagonal of \(\mX^\top\mX\), guaranteeing invertibility even when \(p>n\) or predictors are collinear — ridge always has a unique solution. It shrinks coefficients smoothly toward zero (never exactly to zero), sharing weight among correlated predictors rather than arbitrarily picking one. Probabilistically, ridge is the MAP estimate under a Gaussian prior \(\beta_j\sim\Normal(0,\tau^2)\) — regularization is a prior belief that effects are small.

Lasso (\(L_1\))

The lasso penalizes the \(L_1\) norm, \(\Omega(\vbeta)=\sum_j|\beta_j|\). It has no closed form but is convex and solved efficiently (coordinate descent, LARS).

The lasso’s signature property is sparsity: it drives some coefficients exactly to zero, performing variable selection and fitting simultaneously. The geometry is the reason — the \(L_1\) constraint region is a diamond with sharp corners on the axes, so the elliptical contours of the loss tend to first touch it at a corner, where some coordinates are zero. Lasso is the MAP estimate under a Laplace prior.

Elastic net

The elastic net blends both penalties, \(\Omega(\vbeta)=\alpha\sum_j|\beta_j|+(1-\alpha)\sum_j\beta_j^2\), combining lasso’s sparsity with ridge’s stability.

Pure lasso struggles when predictors are highly correlated — it arbitrarily keeps one and zeros the rest, and it can select at most \(n\) variables when \(p>n\). Elastic net’s ridge component encourages correlated predictors to enter or leave together (the “grouping effect”), making it the practical default for wide, correlated data such as genomics and text.

Always standardize predictors before regularizing. The penalty acts on coefficient magnitudes, so a predictor measured in small units (large coefficient) is penalized more than the same predictor in large units — regularization is not scale-invariant. Also, conventionally the intercept is left unpenalized; you do not want to shrink the baseline level of the response. Most libraries handle both automatically, but verify.

Regularization is everywhere in deep learning, often under the name weight decay — which is precisely the ridge (\(L_2\)) penalty added to the loss, and AdamW made it a default for training Transformers. \(L_1\) penalties induce sparse networks for compression and pruning. The Bayesian reading — penalty \(=\) prior — connects directly to Bayesian neural networks and variational inference. Even techniques without an explicit penalty term, like early stopping and dropout, are understood as implicit regularizers that control the same bias–variance balance. The single equation “loss \(+\,\lambda\cdot\) penalty” is the template for nearly every regularized objective in modern AI.

import numpy as np
from sklearn.linear_model import RidgeCV, LassoCV
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline
rng = np.random.default_rng(0)
n, p = 100, 30
X = rng.normal(size=(n, p))
beta = np.zeros(p); beta[:5] = [3, -2, 1.5, 0, 2]   # only a few truly nonzero
y = X @ beta + rng.normal(0, 1, n)
ridge = make_pipeline(StandardScaler(), RidgeCV(alphas=np.logspace(-2,2,50))).fit(X, y)
lasso = make_pipeline(StandardScaler(), LassoCV(cv=5)).fit(X, y)
print("lasso zeros:", int(np.sum(np.abs(lasso[-1].coef_) < 1e-6)), "of", p)  # sparsity

Chapter summary

  • Regularization minimizes fit \(+\,\lambda\cdot\) penalty, trading a little bias for much less variance; tune \(\lambda\) by cross-validation.

  • Ridge (\(L_2\)) shrinks smoothly, always invertible, shares weight among correlated predictors; it is weight decay and a Gaussian-prior MAP.

  • Lasso (\(L_1\)) yields sparse solutions (exact zeros) — automatic variable selection; Laplace-prior MAP.

  • Elastic net blends both, handling correlated, high-dimensional data with a grouping effect.

  • Standardize first; leave the intercept unpenalized. “Loss + penalty” is the template for regularization across all of ML.