Friedman’s gradient boosting is the idea; XGBoost, LightGBM, and CatBoost are the engineering that made it dominate. Each adds algorithmic and systems innovations — regularized objectives, second-order optimization, histogram binning, clever tree growth, and native categorical handling — that deliver state-of-the-art accuracy on tabular data at remarkable speed. This chapter explains what each brings and how to choose between them.
XGBoost: regularized, second-order boosting
XGBoost (Chen & Guestrin, 2016) sharpens gradient boosting in two ways. First, a regularized objective adds an explicit complexity penalty per tree: \[\mathcal{L} = \sum_i L(y_i,\hat y_i) + \sum_m \Big(\gamma T_m + \tfrac12\lambda\|w_m\|^2\Big),\] penalizing the number of leaves \(T_m\) and the leaf weights \(w_m\). Second, it uses a second-order Taylor expansion of the loss (gradients \(g_i\) and Hessians \(h_i\)), giving a closed-form optimal leaf weight \(w_j^\star=-\frac{\sum_{i\in j}g_i}{\sum_{i\in j}h_i+\lambda}\) and a principled split-gain score. The result: faster convergence, built-in regularization, and support for any twice-differentiable loss.
The second-order view is why XGBoost feels “smarter” than vanilla GBM: instead of just walking downhill on the gradient, it uses curvature (the Hessian) to size each step — a Newton-style update inside every leaf. The \(\gamma\) (minimum split gain) and \(\lambda\) (L2 on weights) terms are literally the cost-complexity pruning idea from Chapter [ch:pruning], now folded into the training objective. XGBoost also adds engineering wins: sparsity-aware split finding (a learned default direction for missing values), cache-aware access, and out-of-core training.
LightGBM: histograms, leaf-wise, GOSS, EFB
LightGBM (Microsoft, 2017) is built for speed on large data:
Histogram binning — bucket continuous features into (e.g.) 255 bins so split finding scans bins, not sorted values: huge memory and time savings.
Leaf-wise growth — grow the leaf with the largest loss reduction (best-first), rather than level-wise; deeper, more accurate trees for the same number of leaves (control with
num_leaves).GOSS (gradient-based one-side sampling) — keep large-gradient (hard) examples, subsample small-gradient ones, to focus computation.
EFB (exclusive feature bundling) — bundle mutually-exclusive sparse features into one, shrinking the effective feature count.
LightGBM is typically the fastest on millions of rows.
CatBoost: ordered boosting and native categoricals
CatBoost (Yandex, 2018) targets categorical features and small-data bias. It encodes categories with ordered target statistics (each row’s encoding uses only prior rows, never its own label) and uses ordered boosting to avoid target leakage / prediction shift — the subtle bias where a model peeks at the target it is trying to predict. It also builds symmetric (oblivious) trees (the same split across an entire level), which act as a regularizer and make inference extremely fast. CatBoost often wins with minimal tuning, especially on categorical-heavy data.
Choosing a library
| XGBoost | LightGBM | CatBoost | |
|---|---|---|---|
| Tree growth | level-wise (depth) | leaf-wise (best-first) | symmetric trees |
| Speed | fast | fastest on big data | fast, fast inference |
| Categoricals | manual encoding | native (integer) | native, ordered |
| Best at | robust all-rounder | large datasets | categorical-heavy, low tuning |
| Watch out | tuning effort | overfits if num_leaves large |
slower training |
Practical traps: (1) LightGBM’s num_leaves is powerful but dangerous — leaf-wise growth overfits if it is large relative to data; keep it \(<2^{\texttt{max\_depth}}\) and use min_child_samples. (2) Don’t one-hot high-cardinality categoricals for LightGBM/CatBoost — use their native handling. (3) Always use early stopping with a validation set rather than guessing n_estimators. (4) Tune learning rate with number of trees together; lower LR + more trees + early stopping is the reliable recipe. (5) Set seeds and log versions — results differ across libraries and versions.
These three libraries are the state of the art for tabular machine learning in industry and competitions — the practical answer to “what model should I try first on a table?” Their innovations also cross-pollinate with deep learning: histogram binning resembles feature discretization in tabular nets; the regularized, second-order objective mirrors modern optimizers; and GBDTs are constantly benchmarked against (and combined with) tabular transformers like FT-Transformer and foundation models like TabPFN (Chapters [ch:vsworld], [ch:treemlai]). Mastering them is, for tabular AI, as fundamental as knowing SGD is for deep learning.
# pip install xgboost lightgbm catboost
import xgboost as xgb
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
X, y = make_classification(n_samples=5000, n_informative=10, random_state=0)
Xtr, Xva, ytr, yva = train_test_split(X, y, random_state=0)
clf = xgb.XGBClassifier(n_estimators=2000, learning_rate=0.03, max_depth=5,
subsample=0.8, colsample_bytree=0.8,
early_stopping_rounds=50, eval_metric="logloss")
clf.fit(Xtr, ytr, eval_set=[(Xva, yva)], verbose=False)
print("best #trees:", clf.best_iteration, " val acc:", round(clf.score(Xva, yva), 3))Chapter summary
XGBoost: regularized objective (\(\gamma T+\tfrac12\lambda\|w\|^2\)) and second-order (gradient + Hessian) optimization with sparsity-aware splits.
LightGBM: histogram binning, leaf-wise growth, GOSS and EFB — fastest on large data (mind
num_leaves).CatBoost: ordered boosting + ordered target encoding to kill prediction shift; native categoricals; symmetric trees.
Pick by data: XGBoost (robust default), LightGBM (large data), CatBoost (many categoricals, low tuning).
Always use early stopping; tune learning rate and tree count together — these libraries are the tabular state of the art.