A compact reference for the formulas, kernels, and knobs used throughout the book.
The SVM at a glance
| Object | Expression |
|---|---|
| Decision rule | \(\hat y=\sgn(\vw^\top\vx+b)\) |
| Margin width | \(2/\|\vw\|\) |
| Hard-margin primal | \(\min \tfrac12\|\vw\|^2\) s.t. \(y_i(\vw^\top\vxi+b)\ge1\) |
| Soft-margin primal | \(\min \tfrac12\|\vw\|^2 + C\sum_i\xi_i\) s.t. \(y_i(\vw^\top\vxi+b)\ge1-\xi_i,\ \xi_i\ge0\) |
| Hinge-loss form | \(\min \tfrac12\|\vw\|^2 + C\sum_i\max(0,1-y_i(\vw^\top\vxi+b))\) |
| Dual | \(\max_\alpha \sum_i\alpha_i-\tfrac12\sum_{i,j}\alpha_i\alpha_j y_iy_j k(\vxi,\vx_j)\), \(0\le\alpha_i\le C\), \(\sum_i\alpha_iy_i=0\) |
| Weight from dual | \(\vw=\sum_i\alpha_i y_i\vxi\) |
| Kernel prediction | \(\hat y(\vx)=\sgn\big(\sum_{i\in\text{SV}}\alpha_i y_i k(\vxi,\vx)+b\big)\) |
| KKT (compl. slack.) | \(\alpha_i[y_i(\vw^\top\vxi+b)-1+\xi_i]=0\) |
Kernel reference
| Kernel | \(k(\vx,\vx')\) | Notes |
|---|---|---|
| Linear | \(\vx^\top\vx'\) | text, \(p\gg n\); fast, interpretable |
| Polynomial | \((\gamma\,\vx^\top\vx'+r)^d\) | degree-\(d\) interactions; tune \(d,\gamma,r\) |
| RBF / Gaussian | \(\exp(-\gamma\|\vx-\vx'\|^2)\) | default nonlinear; tune \(\gamma,C\) |
| Sigmoid | \(\tanh(\gamma\,\vx^\top\vx'+r)\) | not always PSD; use with care |
| String / graph | shared substrings / walks | structured (sequences, molecules) |
Hyperparameter cheat-sheet
| Parameter | Effect | Typical range |
|---|---|---|
C |
inverse regularization; large \(=\) fit hard, narrow margin | \(10^{-2}\)–\(10^{3}\) (log grid) |
gamma (RBF) |
inverse bandwidth; large \(=\) wiggly, overfits | \(10^{-4}\)–\(10^{1}\), or "scale" |
degree (poly) |
polynomial degree | 2–4 |
epsilon (SVR) |
tube half-width (units of \(y\)) | set per target scale |
nu (\(\nu\)-SVM) |
bounds fraction of SV/outliers | \((0,1]\) |
class_weight |
reweight classes for imbalance | "balanced" |
kernel |
similarity function | linear / rbf / poly |
Decision: which solver / model
| Regime | Use |
|---|---|
| Kernel SVM, \(n\lesssim10^5\) | SVC/SVR (LIBSVM, SMO) |
| Linear SVM, large \(n\) | LinearSVC (LIBLINEAR) or SGDClassifier(loss="hinge") |
| Approximate kernel, large \(n\) | random Fourier features / Nyström \(+\) linear SVM |
| Probabilities needed | probability=True (Platt) or calibrate separately |
| Anomaly detection | OneClassSVM (\(\nu\)) |
Key references
Boser, Guyon, Vapnik, “A Training Algorithm for Optimal Margin Classifiers” (1992).
Cortes & Vapnik, “Support-Vector Networks,” Machine Learning (1995).
Vapnik, The Nature of Statistical Learning Theory (1995); Statistical Learning Theory (1998).
Platt, “Sequential Minimal Optimization” (1998); “Probabilistic Outputs for SVMs” (1999).
Schölkopf & Smola, Learning with Kernels (2002); Shawe-Taylor & Cristianini, Kernel Methods (2004).
Schölkopf et al., “Estimating the Support of a High-Dimensional Distribution” (one-class SVM, 2001).
Shalev-Shwartz et al., “Pegasos: Primal Estimated sub-GrAdient SOlver for SVM” (2007).
Jacot et al., “Neural Tangent Kernel” (2018); Chen et al., “On the Equivalence between Neural Network and Support Vector Machine” (2021).
Hastie, Tibshirani, Friedman, The Elements of Statistical Learning; James et al., An Introduction to Statistical Learning.