The maximum-margin idea is not limited to classification. Support Vector Regression (SVR) flips the logic: instead of keeping points outside a margin, it tries to keep them inside a tube around the prediction, ignoring small errors entirely. The result is a flexible, kernelizable, sparse regressor with a distinctive robustness to noise.

The \(\epsilon\)-insensitive tube

SVR fits a function \(f(\vx)=\vw^\top\vx+b\) (or its kernelized version) and penalizes only errors larger than a tolerance \(\epsilon\). Inside the \(\epsilon\)-tube (\(|y-f(\vx)|\le\epsilon\)) the loss is zero; outside, it grows linearly. The objective mirrors the classification SVM: \[\min_{\vw,b,\xi,\xi^*}\ \tfrac12\|\vw\|^2 + C\sum_i(\xi_i+\xi_i^*)\] subject to the residuals lying within \(\epsilon\) plus slacks \(\xi_i,\xi_i^*\ge0\) above/below the tube.

The \(\epsilon\)-insensitive loss is \(\ell_\epsilon(y,f) = \max(0,\ |y-f|-\epsilon)\). It is the regression analog of the hinge loss: flat (zero) in a band, linear beyond it.

Support vectors in regression

As with classification, the KKT conditions make SVR sparse: points strictly inside the tube have \(\alpha_i=0\) and are ignored; only points on or outside the tube boundary become support vectors and define \(f\). So \(\epsilon\) controls sparsity and tube width, while \(C\) controls how hard tube violations are penalized. A wider \(\epsilon\) means fewer support vectors and a smoother fit; a larger \(C\) tracks the data more tightly.

Kernelized SVR and \(\nu\)-SVR

Everything kernelizes: replace inner products with \(k(\vxi,\vx_j)\) and SVR fits smooth nonlinear curves, \(f(\vx)=\sum_{i\in\text{SV}}(\alpha_i-\alpha_i^*)\,k(\vxi,\vx)+b\). A useful variant, \(\nu\)-SVR, replaces the unintuitive \(\epsilon\) with a parameter \(\nu\in(0,1]\) that directly upper-bounds the fraction of points allowed outside the tube and lower-bounds the fraction of support vectors — often easier to set than \(\epsilon\).

Why the tube matters

The flat-bottomed loss gives SVR a robustness that squared-error regression lacks: small residuals cost nothing, so the fit is not pulled around by every minor wiggle, and the linear (not quadratic) growth outside the tube limits the influence of large outliers. The trade-off is three coupled hyperparameters (\(C\), \(\epsilon\) or \(\nu\), and kernel parameters) and the same \(O(n^2)\) scaling as kernel classification.

SVR pitfalls echo the classifier’s, plus one of its own: scale the target too. Because \(\epsilon\) is in the units of \(y\), an \(\epsilon\) chosen for targets in the thousands is meaningless for targets in fractions — standardize features and consider scaling \(y\). Other traps: setting \(\epsilon\) too large (the model predicts a near-constant), too small (every point becomes a support vector, slow and overfit), and forgetting that SVR, like all kernel methods, struggles past \(\sim10^5\) samples.

The \(\epsilon\)-insensitive loss is part of a family of robust regression losses that pervade modern ML — the Huber loss in robust statistics and the smooth-\(L_1\) loss used for bounding-box regression in object detectors (Fast R-CNN, SSD, YOLO) share its “flat near zero, linear in the tail” shape. Kernel ridge regression and Gaussian process regression are close cousins of SVR in the kernel-methods family, and remain go-to tools for small-data regression with uncertainty. The principle “don’t penalize errors you don’t care about” recurs whenever we design losses tolerant to noise.

import numpy as np
from sklearn.svm import SVR
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
rng = np.random.default_rng(0)
X = np.sort(rng.uniform(0, 10, 200)).reshape(-1, 1)
y = np.sin(X).ravel() + rng.normal(0, 0.1, 200)
model = make_pipeline(StandardScaler(), SVR(kernel="rbf", C=10, gamma=0.5, epsilon=0.1))
model.fit(X, y)
print("R^2:", round(model.score(X, y), 3))
print("# support vectors:", model[-1].support_.size, "of", len(X))

Chapter summary

  • SVR fits a function with an \(\epsilon\)-insensitive tube: errors within \(\pm\epsilon\) cost nothing, beyond it grow linearly.

  • The objective \(\tfrac12\|\vw\|^2 + C\sum(\xi_i+\xi_i^*)\) mirrors classification; \(\epsilon\) sets tube width/sparsity, \(C\) sets the penalty.

  • Only points on/outside the tube are support vectors; SVR is sparse and kernelizable for nonlinear fits.

  • \(\nu\)-SVR swaps \(\epsilon\) for \(\nu\), bounding the fraction of outliers/support vectors directly.

  • Scale features and target; the \(\epsilon\)-insensitive loss is kin to the smooth-\(L_1\) losses used in modern detectors.