Now we make the widest-street idea precise. For data that can be perfectly separated by a line, the maximal margin classifier (the hard-margin SVM) is the cleanest version of the story: a little geometry turns “make the street as wide as possible” into a concrete optimization problem whose solution depends only on the support vectors.

Hyperplanes and the geometry of the margin

A hyperplane in \(\R^p\) is the set \(\{\vx : \vw^\top\vx + b = 0\}\). The vector \(\vw\) is normal (perpendicular) to it. The signed distance from a point \(\vx_0\) to the hyperplane is \[\frac{\vw^\top\vx_0 + b}{\|\vw\|}.\]

The sign of \(\vw^\top\vx+b\) says which side of the boundary you are on; the magnitude (divided by \(\|\vw\|\)) says how far. So if we label classes \(y\in\{-1,+1\}\), the quantity \(y_i(\vw^\top\vxi+b)\) is positive exactly when point \(i\) is on the correct side, and its size measures the margin of that point. This is why the labels are chosen as \(\pm1\) rather than \(0/1\) — it makes “correct and confident” a single clean inequality.

The canonical form and the width of the street

We are free to rescale \(\vw,b\), so we fix the scale by requiring the closest points to satisfy \(y_i(\vw^\top\vxi+b)=1\). The two margin surfaces are then \(\vw^\top\vx+b=\pm1\), and the distance between them — the width of the street — works out to \[\text{margin} = \frac{2}{\|\vw\|}.\] Maximizing the margin is therefore the same as minimizing \(\|\vw\|\) (equivalently \(\tfrac12\|\vw\|^2\)).

The hard-margin optimization problem

The maximal margin (hard-margin) classifier solves \[\min_{\vw,\,b}\ \tfrac12\|\vw\|^2 \quad\text{subject to}\quad y_i(\vw^\top\vxi + b)\ \ge\ 1\ \ \text{for all } i.\] This is a convex quadratic program: a quadratic objective with linear constraints, so it has a unique global optimum.

Read the problem in plain words: “make the weight vector as short as possible (widest street) while keeping every point at least one full margin onto its correct side.” Convexity matters enormously — unlike training a neural network, there are no local minima or random restarts; the same data always yields the same optimal boundary. The solution turns out to be a weighted combination of just the support vectors (Chapter [ch:duality]).

Support vectors

At the optimum, points fall into two groups. Support vectors sit exactly on the margin (\(y_i(\vw^\top\vxi+b)=1\)) and determine the boundary. All other points are strictly beyond the margin and are irrelevant: delete them and re-train, and you get the same boundary. This is the SVM’s defining sparsity — the model is summarized by a few critical examples, not the whole dataset.

The hard-margin classifier has two fatal flaws on real data: (1) it requires perfect linear separability — if even one point is on the wrong side, the constraints are infeasible and there is no solution; and (2) it is exquisitely sensitive to outliers — a single mislabeled point can swing the boundary wildly or shrink the margin to nothing. Real data is noisy and overlapping, so we almost never use hard margins directly. The fix — allowing a budget of violations — is the soft-margin SVM of the next chapter.

The move “\(\text{maximize margin}\Leftrightarrow\text{minimize }\|\vw\|^2\)” is the first appearance of \(L_2\) regularization in this book, and it is no coincidence: penalizing \(\|\vw\|^2\) is exactly weight decay in neural networks and ridge regression in statistics. The SVM reveals the geometric meaning of that penalty — shorter weights mean wider margins mean more robust decisions. The same principle, “prefer the flattest function that fits,” echoes through every regularized model in modern AI.

import numpy as np
from sklearn.svm import SVC
# Perfectly separable toy data
X = np.array([[1,1],[2,1],[1,2],[5,5],[6,5],[5,6]])
y = np.array([-1,-1,-1, 1, 1, 1])
clf = SVC(kernel="linear", C=1e6).fit(X, y)        # huge C ~ hard margin
w, b = clf.coef_[0], clf.intercept_[0]
print("margin width 2/||w|| =", round(2/np.linalg.norm(w), 3))
print("support vectors:\n", clf.support_vectors_)   # only the boundary points

Chapter summary

  • A hyperplane \(\vw^\top\vx+b=0\) has normal \(\vw\); the signed distance of a point is \((\vw^\top\vx+b)/\|\vw\|\).

  • With labels \(y\in\{\pm1\}\) and the canonical scaling, the margin width is \(2/\|\vw\|\).

  • Maximizing the margin \(\Leftrightarrow\) minimizing \(\tfrac12\|\vw\|^2\) subject to \(y_i(\vw^\top\vxi+b)\ge1\) — a convex QP.

  • Only support vectors (points on the margin) determine the boundary; the rest are irrelevant.

  • Hard margins fail on non-separable or noisy data and are outlier-sensitive — motivating soft margins next.