Classical (“tabular”) machine learning is still where most production value is created outside of the LLM frontier: churn prediction, fraud, pricing, demand forecasting, ranking. It is also the best place to build the disciplines — evaluation, validation, leakage avoidance — that separate practitioners from people who overfit Kaggle leaderboards. Deep learning inherits all of these disciplines; learning them on simple models first is far cheaper.

ML fundamentals

The learning setups

• Supervised: learn a mapping \(f:\vx\mapsto y\) from labeled pairs. Regression (\(y\in\R\)) or classification (\(y\) categorical).

• Unsupervised: find structure without labels — clustering, dimensionality reduction, density estimation.

• Semi-supervised: a little labeled data plus a lot of unlabeled data (common and economically important).

• Reinforcement: an agent learns from reward by interacting with an environment (Phase 6).

Train / validation / test, and cross-validation

You split data to estimate generalization — performance on data the model has never seen. The cardinal rule: the test set is touched exactly once, at the very end.

Bias–variance tradeoff

Expected test error decomposes (for squared loss) into three parts: \[\E\big[(y-\hat f(\vx))^2\big] = \underbrace{\big(\text{Bias}[\hat f(\vx)]\big)^2}_{\text{too simple}} + \underbrace{\Var[\hat f(\vx)]}_{\text{too sensitive}} + \underbrace{\sigma^2}_{\text{irreducible}}.\]

Evaluation metrics — choose before you model

Regression

MSE \(=\frac1n\sum(y_i-\hat y_i)^2\) (penalizes large errors quadratically); RMSE \(=\sqrt{\text{MSE}}\) (same units as \(y\)); MAE \(=\frac1n\sum\abs{y_i-\hat y_i}\) (robust to outliers); \(R^2=1-\frac{\sum(y_i-\hat y_i)^2}{\sum(y_i-\bar y)^2}\) (fraction of variance explained).

Classification

From the confusion matrix (TP, FP, TN, FN): \[\text{precision}=\frac{TP}{TP+FP},\quad \text{recall}=\frac{TP}{TP+FN},\quad F_1=\frac{2\cdot\text{prec}\cdot\text{rec}}{\text{prec}+\text{rec}}.\] ROC-AUC measures ranking quality across all thresholds (probability a random positive outranks a random negative). PR-AUC is more informative under heavy class imbalance.

Ranking

For search/recommendation, NDCG (normalized discounted cumulative gain) rewards putting relevant items high; MRR (mean reciprocal rank) focuses on the position of the first relevant result. These matter the moment your API powers a search or recommendation feature.

Regression & classification

Linear regression — two solvers

The model is \(\hat y = \vw\tp\vx + b\). With the bias folded into \(\vw\) by appending a 1 to \(\vx\):

• Closed form (normal equation): \(\vw=(\mX\tp\mX)^{-1}\mX\tp\vy\) — exact, but \(O(d^3)\) in the number of features and unstable if \(\mX\tp\mX\) is ill-conditioned.

• Gradient descent: scales to huge \(n\) and \(d\), and generalizes to models with no closed form. (You implemented this in Phase 0.)

Regularization: Ridge, Lasso, ElasticNet

Add a penalty on weight magnitude to combat overfitting and multicollinearity: \[\text{Ridge: } \min_\vw \norm{\mX\vw-\vy}_2^2 + \lambda\norm{\vw}_2^2,\qquad \text{Lasso: } \min_\vw \norm{\mX\vw-\vy}_2^2 + \lambda\norm{\vw}_1.\]

Logistic and softmax regression

For binary classification, logistic regression models \(\Prob(y=1\mid\vx)=\sigma(\vw\tp\vx)\) with the sigmoid \(\sigma(z)=1/(1+e^{-z})\). It is trained by minimizing the cross-entropy (negative log-likelihood of the Bernoulli): \[L(\vw)=-\sum_i\big[y_i\log\hat p_i+(1-y_i)\log(1-\hat p_i)\big],\quad \hat p_i=\sigma(\vw\tp\vx_i).\] For \(K\) classes, softmax regression outputs \(\hat p_k=\frac{e^{z_k}}{\sum_j e^{z_j}}\) and minimizes categorical cross-entropy. This exact pairing (softmax + cross-entropy) is the output layer of nearly every classifier you will build in Phases 2–4.

Naive Bayes

Apply Bayes’ theorem with the (“naive”) assumption that features are conditionally independent given the class: \(\Prob(y\mid\vx)\propto \Prob(y)\prod_j \Prob(x_j\mid y)\). Despite the unrealistic assumption it is fast, needs little data, and is a strong baseline for text classification (bag-of-words).

K-Nearest Neighbors

KNN predicts using the \(k\) closest training points (majority vote or average). It has no training phase (lazy learning) but expensive inference, and degrades in high dimensions (the curse of dimensionality). It is a useful conceptual anchor: it makes the role of distance metrics and feature scaling vivid.

Tree-based methods

Decision trees

A tree recursively splits the feature space to make regions as “pure” as possible. Split quality for classification uses Gini impurity \(G=1-\sum_k p_k^2\) or entropy \(H=-\sum_k p_k\log p_k\); the chosen split maximizes information gain (impurity reduction). Trees are interpretable and need no feature scaling, but a single deep tree overfits badly (high variance).

Random forests (bagging)

Bagging (bootstrap aggregating) trains many trees on bootstrap resamples and averages them, slashing variance. Random forests add a twist: at each split only a random subset of features is considered, decorrelating the trees so averaging helps more. Robust, low-tuning, excellent default for tabular data.

Gradient boosting

Boosting builds trees sequentially, each new tree fitting the residual errors of the ensemble so far — gradient descent in function space.

Interpretability: feature importance & SHAP

Built-in importances (split gain) are quick but biased toward high-cardinality features. SHAP values, grounded in cooperative game theory, fairly attribute each prediction to its features and are consistent and locally accurate. SHAP summary and dependence plots are the modern standard for explaining tabular models to stakeholders — and for catching leakage (a single feature with implausibly dominant SHAP is a red flag).

Support Vector Machines

An SVM finds the hyperplane that maximizes the margin — the distance to the nearest points (support vectors). The soft-margin formulation allows some violations, controlled by \(C\) (large \(C\) = fewer violations, higher variance). The kernel trick replaces dot products \(\vx_i\tp\vx_j\) with a kernel \(k(\vx_i,\vx_j)\), implicitly mapping to a high-dimensional space without computing it — enabling nonlinear boundaries with linear, polynomial, or RBF kernels.

Unsupervised learning

Clustering

K-Means alternates assigning points to the nearest centroid and recomputing centroids; it minimizes within-cluster variance but assumes spherical, equal-size clusters and needs \(k\) chosen (elbow/silhouette methods). Hierarchical clustering builds a dendrogram (no preset \(k\)). DBSCAN finds density-connected clusters of arbitrary shape and labels outliers, without specifying \(k\) — but is sensitive to its density parameters.

Dimensionality reduction

PCA projects onto the top eigenvectors of the covariance matrix (the SVD of centered data) — a linear method that maximizes retained variance, great for compression and de-noising. t-SNE and UMAP are nonlinear methods for visualization of clusters in 2D/3D.

Anomaly detection

Isolation Forest isolates outliers with random splits (anomalies need fewer splits); One-Class SVM learns a boundary around the normal data. Both are unsupervised and directly relevant to fraud/defect/shipment-anomaly use cases.

Feature engineering & data prep

In tabular ML this is where most of the winning happens.

• Missing data: understand why it is missing; impute (mean/median/model-based) and optionally add a “was-missing” indicator.

• Outliers: detect (IQR, z-score), then cap/winsorize or model robustly — do not blindly delete.

• Categorical encoding: one-hot for low cardinality; target/mean encoding for high cardinality (with cross-fold smoothing to avoid leakage); ordinal only when order is real.

• Scaling: standardize (\(z\)-score) or normalize (\([0,1]\)). Required for KNN, SVM, PCA, and neural nets; irrelevant for trees.

• Class imbalance: class weighting, undersampling the majority, or SMOTE (synthetic minority oversampling). Apply resampling inside cross-validation only.

from sklearn.pipeline import Pipeline
from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.impute import SimpleImputer
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.model_selection import cross_val_score

num = Pipeline([("imp", SimpleImputer(strategy="median")),
                ("sc", StandardScaler())])
cat = Pipeline([("imp", SimpleImputer(strategy="most_frequent")),
                ("oh", OneHotEncoder(handle_unknown="ignore"))])
pre = ColumnTransformer([("num", num, num_cols), ("cat", cat, cat_cols)])

model = Pipeline([("pre", pre), ("clf", HistGradientBoostingClassifier())])
# Transforms are re-fit on each training fold -> no leakage.
scores = cross_val_score(model, X, y, cv=5, scoring="roc_auc")
print(scores.mean(), scores.std())

Model selection & tuning

• Grid search exhaustively tries combinations (expensive, fine for few hyperparameters).

• Random search samples combinations — usually more efficient than grid for the same budget.

• Bayesian optimization (Optuna) models the objective and proposes promising points; best for expensive models and larger search spaces. Use pruning to kill bad trials early.

Learning vs. validation curves

Learning curves plot train/validation score vs. training-set size: a large persistent gap signals high variance (get more data / regularize); both curves low and converged signals high bias (use a richer model / better features). Validation curves plot score vs. a single hyperparameter to find its sweet spot.

A/B testing for production models

Offline metrics do not always predict online impact. The gold standard is a randomized A/B test: split traffic, run the new model against the incumbent, and compare a pre-registered business metric with proper statistics (mind the multiple-comparisons and peeking problems from Phase 0). This bridges directly to Phase 7.

Checkpoint project

Phase 1 self-check

• Choose and justify an evaluation metric for an imbalanced business problem.

• Explain bias–variance using a learning curve you drew.

• State when Ridge vs. Lasso vs. ElasticNet is appropriate and why.

• Explain why gradient boosting usually beats a single tree and a random forest on tabular data.

• Build a cross-validated, leakage-free pipeline and explain every transform’s placement.

• Read a SHAP summary plot and identify a suspicious (leaky) feature.