When the relationship is genuinely curved and you do not know its form, you need methods that let the data dictate the shape. This chapter surveys the flexible end of regression: polynomials and splines, local and kernel regression, generalized additive models, regression trees and forests, and Gaussian processes. These are the bridge from the rigid linear model to the universal function approximators of deep learning.

Polynomials and splines

A high-degree polynomial can fit any wiggle, but it oscillates wildly between points and explodes at the boundaries (Runge’s phenomenon). Splines fix this by stitching together low-degree (usually cubic) polynomial pieces at knots, forced to join smoothly. Regression splines place a handful of knots; smoothing splines put a knot at every point but add a roughness penalty \(\lambda\int f''(x)^2\,dx\) — regularization (Chapter [ch:regularization]) reappearing to control wiggliness.

Local and kernel regression

Local regression (LOESS/LOWESS) estimates \(\hat f(x_0)\) by fitting a low-degree polynomial to the points near \(x_0\), weighted by distance through a kernel. Kernel regression (Nadaraya–Watson) is the simplest case: a locally weighted average, \[\hat f(x_0)=\frac{\sum_i K_h(x_0-x_i)\,y_i}{\sum_i K_h(x_0-x_i)},\] with bandwidth \(h\) setting the neighborhood size.

The bandwidth \(h\) is the bias–variance dial in disguise: small \(h\) follows the data closely (low bias, high variance, jagged); large \(h\) smooths heavily (high bias, low variance, flat). These methods are nonparametric — they keep all the training data and have no fixed set of coefficients, so model complexity grows with the data rather than being fixed in advance.

Generalized additive models

A GAM keeps the additive, interpretable structure of linear regression but replaces each linear term with a smooth function: \[y=\beta_0+f_1(x_1)+f_2(x_2)+\dots+f_p(x_p)+\varepsilon,\] where each \(f_j\) is typically a spline. You see each predictor’s individual nonlinear effect while avoiding the curse of dimensionality of a full \(p\)-dimensional smoother.

Trees, forests, and boosting

A regression tree recursively splits the predictor space into boxes and predicts the mean response in each — a piecewise-constant surface, completely nonparametric and able to capture interactions automatically. A single tree is high-variance, so we average many: random forests (bagging decorrelated trees) and gradient boosting (adding trees that fit the residuals of the current ensemble) are among the most accurate off-the-shelf regressors for tabular data, routinely beating both linear models and neural networks there.

Gaussian process regression

Gaussian process (GP) regression is the Bayesian nonparametric ideal: instead of a single fitted curve, it places a distribution over all smooth functions consistent with the data, yielding a predictive mean and calibrated uncertainty that widens away from observed points. A kernel encodes assumed smoothness; the posterior is computed in closed form via a Cholesky solve. GPs are the workhorse of Bayesian optimization (used to tune ML hyperparameters), where knowing where the model is uncertain is as valuable as the prediction itself.

Flexibility is a double-edged sword. Nonparametric methods overfit eagerly — their smoothing parameters (\(h\), knot count, tree depth, kernel scale) must be set by cross-validation, not eyeballed. They also suffer the curse of dimensionality: local methods and kernels degrade badly as \(p\) grows because “nearby” points become impossibly sparse. And kernel/GP methods cost \(O(n^2)\)–\(O(n^3)\), limiting them to modest \(n\) without approximations. For high-dimensional, large-\(n\) problems, structured models (trees) or neural networks usually win.

This chapter is the conceptual on-ramp to deep learning. Neural networks are flexible nonparametric regressors — the universal approximation theorem says a sufficiently wide network can approximate any continuous function, just like splines or kernels, but they scale to high dimensions where classical nonparametrics fail. The connections are deep and literal: a wide random network converges to a Gaussian process (the Neural Tangent Kernel); the kernel trick that powers kernel regression also powers support vector regression; and gradient-boosted trees remain the state of the art for tabular regression, often outperforming deep nets. The trade-off you have seen — flexibility versus overfitting, controlled by a smoothing knob and cross-validation — is exactly how we reason about network capacity.

import numpy as np
from sklearn.ensemble import RandomForestRegressor
from sklearn.gaussian_process import GaussianProcessRegressor
from sklearn.gaussian_process.kernels import RBF
rng = np.random.default_rng(0)
X = np.sort(rng.uniform(-3, 3, 60)).reshape(-1, 1)
y = np.sin(X).ravel() + rng.normal(0, 0.1, 60)
rf = RandomForestRegressor(n_estimators=200, random_state=0).fit(X, y)
gp = GaussianProcessRegressor(kernel=RBF(1.0), alpha=0.01).fit(X, y)
mean, std = gp.predict(X, return_std=True)   # GP gives mean AND uncertainty
print("RF R^2:", round(rf.score(X, y), 3), " GP mean std:", round(std.mean(), 3))

Chapter summary

  • Splines stitch smooth polynomial pieces; smoothing splines add a roughness penalty (regularization again).

  • Local/kernel regression averages nearby points; the bandwidth \(h\) is the bias–variance dial.

  • GAMs keep additivity with smooth per-predictor effects; trees/forests/boosting are top tabular regressors.

  • Gaussian processes give a posterior over functions with calibrated uncertainty — the engine of Bayesian optimization.

  • All are flexible nonparametric regressors needing CV-tuned smoothing; neural nets extend this to high dimensions (universal approximation, NTK).