A compact reference to the book’s key formulas and a decision guide for choosing a method.
Core formulas
| Quantity | Formula |
|---|---|
| Simple slope / intercept | \(\hat\beta_1=\dfrac{\Cov(x,y)}{\Var(x)},\quad \hat\beta_0=\bar y-\hat\beta_1\bar x\) |
| Residual sum of squares | \(\RSS=\sum_i (y_i-\hat y_i)^2\) |
| Coefficient of determination | \(R^2=1-\RSS/\TSS,\quad \TSS=\sum_i(y_i-\bar y)^2\) |
| Adjusted \(R^2\) | \(1-(1-R^2)\dfrac{n-1}{n-p-1}\) |
| Normal equations (OLS) | \(\bhat=(\mX^\top\mX)^{-1}\mX^\top\vy\) |
| Hat (projection) matrix | \(\mH=\mX(\mX^\top\mX)^{-1}\mX^\top,\quad \vyhat=\mH\vy\) |
| Coefficient covariance | \(\widehat{\Cov}(\bhat)=\hat\sigma^2(\mX^\top\mX)^{-1}\) |
| Ridge estimate | \(\bhat_{\text{ridge}}=(\mX^\top\mX+\lambda\mathbf I)^{-1}\mX^\top\vy\) |
| Lasso objective | \(\min_{\vbeta}\ \norm{\vy-\mX\vbeta}^2+\lambda\sum_j|\beta_j|\) |
| Elastic net penalty | \(\lambda\big[\alpha\sum_j|\beta_j|+(1-\alpha)\sum_j\beta_j^2\big]\) |
| Logistic model | \(p(\vx)=\sigma(\vbeta^\top\vx),\ \ \sigma(z)=1/(1+e^{-z})\) |
| Binary cross-entropy | \(-\sum_i[y_i\ln p_i+(1-y_i)\ln(1-p_i)]\) |
| Softmax | \(\Pr(y=k\mid\vx)=e^{\vbeta_k^\top\vx}\big/\sum_j e^{\vbeta_j^\top\vx}\) |
| GLM link | \(g(\mu)=\vbeta^\top\vx,\quad \mu=g^{-1}(\vbeta^\top\vx)\) |
| VIF | \(\mathrm{VIF}_j=1/(1-R_j^2)\) |
| Bias–variance | \(\E[(y-\hat f)^2]=\text{Bias}^2+\Var+\sigma^2\) |
| AIC / BIC | \(-2\ln\hat L+2k\ \ /\ \ -2\ln\hat L+k\ln n\) |
| Huber loss | quadratic for \(|e|\le\delta\), linear beyond |
| Pinball (quantile) loss | \(\rho_\tau(e)=\max(\tau e,(\tau-1)e)\) |
Which method should I use?
| Situation | Recommended method |
|---|---|
| Continuous \(y\), few predictors | ordinary least squares (Ch. [ch:simple], [ch:multiple]) |
| Many / correlated predictors | ridge; lasso/elastic net for selection (Ch. [ch:regularization]) |
| \(p>n\) (wide data) | ridge or elastic net (Ch. [ch:regularization]) |
| Binary outcome | logistic regression (Ch. [ch:logistic]) |
| Multiclass outcome | softmax / multinomial logistic (Ch. [ch:logistic]) |
| Count outcome | Poisson / negative-binomial GLM (Ch. [ch:glm]) |
| Positive, skewed outcome | Gamma GLM or log-transform (Ch. [ch:glm]) |
| Curved relationship, known form | polynomial / basis features (Ch. [ch:features]) |
| Curved, unknown form, low-dim | splines, GAMs, LOESS (Ch. [ch:nonparametric]) |
| Tabular, high accuracy | gradient boosting / random forest (Ch. [ch:nonparametric]) |
| Need uncertainty / small data | Gaussian process, Bayesian regression (Ch. [ch:nonparametric], [ch:robust]) |
| Outliers present | Huber / least-absolute-deviations (Ch. [ch:robust]) |
| Need intervals / tails | quantile regression (Ch. [ch:robust]) |
| High-dim, large \(n\), complex | neural network (Ch. [ch:regmlai]) |
Modeling checklist
Define the goal: prediction or inference? Choose the loss/metric accordingly.
Split off a test set before any exploration; build features in a leak-free pipeline.
Fit a simple linear baseline; standardize predictors before regularizing.
Check residual plots (linearity, equal variance, normality, outliers, leverage).
Tune complexity / \(\lambda\) by cross-validation; prefer the simplest model within one SE of the best.
Report the chosen metric with uncertainty, against a baseline, on the untouched test set.
State assumptions explicitly; do not read coefficients causally without a design that licenses it.
Symbols
| Symbol | Meaning |
|---|---|
| \(y,\ \hat y\) | response; fitted/predicted value |
| \(\vx,\ \mX\) | predictor vector; design matrix (\(n\times(p+1)\)) |
| \(\vbeta,\ \bhat\) | coefficient vector; its estimate |
| \(\varepsilon,\ e\) | error term; residual \(y-\hat y\) |
| \(\lambda\) | regularization strength |
| \(\sigma^2\) | irreducible noise variance |
| \(\sigma(\cdot)\) | logistic (sigmoid) function |
| \(n,\ p\) | number of observations; number of predictors |