We ended the last chapter with a striking reformulation: the soft-margin SVM is just \(L_2\) regularization plus a special loss. That loss is the hinge loss, and viewing the SVM as “minimize regularized hinge loss” (the primal problem) demystifies it, connects it to logistic regression, and lets us train SVMs by plain gradient descent — the same way we train neural networks.
The hinge loss
The hinge loss for a labeled point \((\vx,y)\) with score \(s=\vw^\top\vx+b\) is \[\ell_{\text{hinge}}(y,s) = \max(0,\ 1 - y\,s).\] It is zero once the point is correct with margin (\(y\,s\ge1\)), and grows linearly as the point moves into or across the margin.
The hinge loss is a convex surrogate for the (discontinuous, intractable) 0–1 misclassification loss. Crucially, it has a flat zero region: points already classified with enough margin contribute nothing to the loss or its gradient. That flat region is precisely why SVMs are sparse — only points on or inside the margin (the support vectors) push on the solution. Compare logistic loss, which is always positive: every point nudges a logistic model, so it has no support vectors.
The primal objective
The (soft-margin) SVM primal problem is unconstrained minimization of regularized hinge loss: \[\min_{\vw,b}\ \ \frac{1}{2}\|\vw\|^2 \ +\ C\sum_{i=1}^n \max\!\big(0,\ 1 - y_i(\vw^\top\vxi+b)\big).\] Equivalently, dividing by \(C\), it is \(\frac{\lambda}{2}\|\vw\|^2 + \frac1n\sum_i \ell_{\text{hinge}}\) with \(\lambda\propto 1/C\) — average hinge loss plus an \(L_2\) penalty. This is a convex, piecewise-quadratic function of \((\vw,b)\) with a unique minimum.
Training by (sub)gradient descent
Because the primal is convex, we can minimize it directly. The hinge loss has a kink at \(ys=1\), so we use a subgradient: \[\nabla_{\vw}\,\ell_i = \begin{cases} \mathbf{0} & \text{if } y_i(\vw^\top\vxi+b)\ge 1\ \ (\text{safe point}),\\[2pt] -\,y_i\,\vxi & \text{otherwise}\ \ (\text{margin violator}). \end{cases}\] The full gradient adds the regularizer’s \(\vw\). A point only contributes a “push” (\(-y_i\vxi\)) when it violates its margin — the update literally drags the boundary toward the troublemakers. Stochastic subgradient descent (e.g. the Pegasos algorithm) trains linear SVMs on huge datasets this way.
Hinge versus logistic
| SVM (hinge) | Logistic regression (log-loss) | |
|---|---|---|
| Loss | \(\max(0,1-ys)\) | \(\log(1+e^{-ys})\) |
| Zero-loss region | yes (\(ys\ge1\)) | no (always \(>0\)) |
| Solution sparsity | support vectors only | uses all points |
| Probabilities | not native (needs calibration) | native \(\sigma(s)\) |
| Robust to outliers | moderately (linear growth) | similar (linear tail) |
Don’t expect probabilities from a raw SVM. The hinge loss optimizes the decision boundary, not calibrated likelihoods; the score \(\vw^\top\vx+b\) is a signed distance, not \(P(y\mid\vx)\). If you need probabilities, fit a calibration map (Platt scaling / isotonic) on held-out data — which is what scikit-learn’s probability=True does internally (Chapter [ch:practical]). Also note the kink: standard gradient descent works fine with subgradients, but “the gradient is undefined at \(ys=1\)” trips up naive implementations.
The hinge loss is not a museum piece — it is a standard tool in deep learning. “Linear-SVM-on-top” networks replace the softmax/cross-entropy head with a linear hinge (L2-SVM) loss and sometimes train better on classification benchmarks. Margin-based and triplet losses for metric learning, face recognition, and retrieval are direct descendants of the hinge’s “correct by a margin or pay” principle. And the primal view — regularizer plus convex surrogate loss, minimized by (stochastic sub)gradient descent — is exactly how neural networks are trained, making the SVM the cleanest gateway to understanding modern optimization.
import numpy as np
from sklearn.linear_model import SGDClassifier # linear SVM via subgradient descent
from sklearn.datasets import make_classification
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline
X, y = make_classification(n_samples=2000, n_features=20, random_state=0)
clf = make_pipeline(StandardScaler(),
SGDClassifier(loss="hinge", alpha=1e-3, max_iter=1000)) # hinge = SVM
clf.fit(X, y)
print("train accuracy:", round(clf.score(X, y), 3))Chapter summary
The hinge loss \(\max(0,1-ys)\) is a convex surrogate for 0–1 loss with a flat zero region beyond the margin.
The SVM primal minimizes \(\tfrac12\|\vw\|^2 + C\sum_i\text{hinge}_i\) — \(L_2\) regularization plus average hinge loss.
The flat region makes the solution sparse (support vectors); logistic loss has no such region.
The primal is convex and trainable by (sub)gradient descent (e.g. Pegasos), scaling to large linear problems.
SVMs don’t output probabilities natively; calibrate if needed. Hinge/margin losses live on throughout deep learning.