Ordinary least squares is optimal under Gaussian noise — but real data has outliers, heavy tails, and asymmetric costs, and sometimes we want full probability distributions over our answers rather than point estimates. This chapter covers three expert extensions: robust regression (resistant to outliers), quantile regression (modeling the whole conditional distribution), and Bayesian regression (coherent uncertainty). Each changes the loss or the inference, not the underlying linear structure.

Robust regression

Because least squares squares residuals, a single far-off point can dominate the fit — one outlier with residual 10 contributes as much as a hundred points with residual 1. Robust methods curb this by penalizing large residuals less aggressively.

Robust regression replaces the squared loss with a loss that grows slowly for large residuals:

  • Least absolute deviations (\(L_1\)): minimize \(\sum_i|e_i|\) — fits the conditional median, far less sensitive to outliers.

  • Huber loss: quadratic for small residuals, linear beyond a threshold \(\delta\) — a smooth compromise that is efficient and robust.

The Huber curve is the practical sweet spot: near zero it behaves like least squares (statistically efficient when noise is Gaussian), but its linear tails stop outliers from hijacking the fit. This exact loss is widely used in machine learning — e.g. the smooth_l1 / Huber loss in object detection and reinforcement learning — precisely for its robustness.

Quantile regression

Ordinary regression models the conditional mean; quantile regression models any conditional quantile \(\tau\) (e.g. the median \(\tau=0.5\), or the 90th percentile) by minimizing the asymmetric pinball loss \[\rho_\tau(e)=\begin{cases}\tau\,e & e\ge 0\\(\tau-1)\,e & e<0\end{cases}.\] Fitting several \(\tau\)’s traces out the full conditional distribution, not just its center — revealing how spread and skew change with the predictors.

Quantile regression answers questions the mean cannot: “what is a pessimistic (10th-percentile) delivery time?” or “how do the top earners, not the average, respond to education?” It needs no distributional assumption and is naturally robust. In ML it underlies prediction intervals and probabilistic forecasting — gradient-boosting and neural models routinely output multiple quantiles to quantify uncertainty.

Bayesian regression

Bayesian linear regression treats the coefficients as random, combining a prior \(p(\vbeta)\) with the likelihood \(p(\vy\mid\mX,\vbeta)\) to obtain a posterior via Bayes’ theorem: \[p(\vbeta\mid\Ldata)\;\propto\;p(\vy\mid\mX,\vbeta)\,p(\vbeta).\] Rather than one \(\bhat\), you get a whole distribution over plausible lines.

With a Gaussian prior and Gaussian likelihood the posterior is Gaussian in closed form, and its mean is exactly the ridge estimate of Chapter [ch:regularization] — regularization is a Gaussian prior, made explicit. The payoff is honest uncertainty: every prediction comes with a posterior predictive distribution whose width reflects both noise and parameter uncertainty, and which widens where data is scarce. Priors also stabilize estimation when data is limited or predictors collinear.

Pitfalls across the three. Robust: most robust losses are non-quadratic, so there is no closed form and the optimization can have local issues; tune the threshold \(\delta\). Quantile: fitted quantile curves for different \(\tau\) can cross (a 90th percentile below the 50th), which is logically impossible — use non-crossing constraints. Bayesian: results depend on the prior, and beyond the conjugate Gaussian case the posterior has no closed form, requiring MCMC or variational approximations that demand convergence checks. None of these is a free lunch — they buy robustness or uncertainty at the cost of computation or assumptions.

All three power modern AI. Huber/smooth-\(L_1\) loss is standard in object detection (Faster R-CNN) and deep reinforcement learning (DQN) for stable training under outliers. Quantile regression is the backbone of probabilistic forecasting and of distributional RL (QR-DQN), where an agent learns the full distribution of returns, not just the mean. Bayesian regression is the conceptual seed of Bayesian deep learning: Bayesian neural networks, variational inference, Monte-Carlo dropout, and Laplace approximations all aim to put a posterior over network weights so the model knows what it does not know — essential for safety-critical AI, active learning, and out-of-distribution detection.

import numpy as np
from sklearn.linear_model import HuberRegressor, QuantileRegressor, BayesianRidge
rng = np.random.default_rng(0)
X = rng.normal(size=(100, 1)); y = 2*X.ravel() + rng.normal(0, 0.5, 100)
y[:5] += 15                                   # inject outliers
print("OLS-ish Huber slope:", HuberRegressor().fit(X, y).coef_[0].round(3))
q90 = QuantileRegressor(quantile=0.9, alpha=0).fit(X, y)   # 90th percentile fit
bayes = BayesianRidge().fit(X, y)
mean, std = bayes.predict(X[:3], return_std=True)          # prediction + uncertainty
print("Bayesian pred std:", std.round(3))

Chapter summary

  • Robust regression (\(L_1\), Huber) resists outliers by penalizing large residuals less than squared loss; Huber is the efficient–robust compromise.

  • Quantile regression models conditional quantiles via the asymmetric pinball loss, giving the whole distribution and prediction intervals.

  • Bayesian regression returns a posterior over coefficients; with Gaussian prior/likelihood its mean is the ridge estimate, plus calibrated predictive uncertainty.

  • Each trades computation or assumptions for robustness or uncertainty — no free lunch.

  • These power Huber loss (detection/RL), distributional RL and probabilistic forecasting, and Bayesian deep learning for “knowing what you don’t know.”