When should you reach for a tree ensemble, and when for something else? This chapter places trees in the broader modeling landscape — against linear models and against deep learning — and explains the inductive biases that make trees so well-suited to tabular data. The punchline, backed by large benchmarks: for tables, tuned gradient boosting is still the model to beat.
Trees vs linear models
Linear/logistic models assume a smooth, additive, roughly monotone relationship; they extrapolate, give signed coefficients, and shine when that assumption holds or data is scarce and high-dimensional (with regularization). Trees make no such assumption: they capture nonlinearity and interactions automatically, ignore feature scaling, and resist outliers — but cannot extrapolate and need more data to be stable. Rule of thumb: simple, linear, well-understood signal \(\to\) linear; complex interactions, mixed messy features \(\to\) trees.
Why trees beat deep nets on tabular data
A landmark benchmark (Grinsztajn et al., 2022) found tree-based models outperform deep nets across most tabular datasets, even after heavy tuning, for three inductive-bias reasons:
Irregular, non-smooth targets. Real tabular target functions have sharp steps and thresholds; trees fit them naturally, while neural nets are biased toward overly smooth functions.
Uninformative features. Tables carry many weak/irrelevant columns; trees ignore them via split selection, whereas MLPs are easily distracted.
Non-rotationally-invariant data. Each tabular column has its own meaning; trees are axis-aligned and respect that, while neural nets’ rotational invariance mixes features and discards this structure.
Add lower compute, no scaling, native missing/categorical handling, and exact SHAP, and trees are the pragmatic winner.
When deep learning wins on tables
The picture is nuanced and shifting (2024–2026). Deep models earn their keep when: data is very large; features are high-cardinality categorical learned as embeddings (recommenders); there is unstructured signal (text, images, sequences) mixed with the table, needing end-to-end learning; you want transfer/foundation models for small data (TabPFN excels there via in-context learning); or you need a single differentiable pipeline. Strong tabular nets (FT-Transformer, SAINT, RealMLP) and foundation models now match or beat GBDTs on parts of recent benchmarks (TabArena) — but usually at higher cost.
A decision guide
| Situation | Reach for |
|---|---|
| New tabular problem, want a strong baseline fast | Gradient boosting (XGBoost/LightGBM/CatBoost) |
| Robust, low-tuning, free OOB + importances | Random forest |
| Small data, smooth signal, interpretability | Regularized linear/logistic, GAM, or EBM |
| Huge data, embeddings, text/image + table | Deep tabular net / multi-modal model |
| Very small data, want strong zero-tuning model | TabPFN (foundation model) |
| Maximum accuracy, competition | Stack GBDTs + a tuned neural net |
Don’t fight the data type. Using a vanilla MLP on a plain 5,000-row table and expecting it to beat tuned XGBoost usually wastes weeks. Equally, don’t force trees onto raw images, audio, or text — those have spatial/sequential structure that CNNs and transformers exploit and trees cannot. And beware benchmark cherry-picking: “method X beats GBDT” often hides dataset selection, tuning asymmetry, or ignored compute cost. Match the model’s inductive bias to your data’s structure.
The tree-vs-deep-learning debate is a debate about inductive bias — the central concept of modern ML. Trees encode “axis-aligned, piecewise-constant, scale-free” priors perfect for tables; CNNs encode locality/translation; transformers encode permutation-equivariant attention for sequences. Understanding why each wins on its home turf is what lets you choose architectures wisely. The next chapter closes the loop: rather than competing, trees and deep learning increasingly combine — the frontier of tabular AI.
from sklearn.datasets import fetch_covtype
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.model_selection import train_test_split
X, y = fetch_covtype(return_X_y=True)
Xtr, Xte, ytr, yte = train_test_split(X[:30000], y[:30000], random_state=0)
for name, m in [("logreg", LogisticRegression(max_iter=200)),
("hist-GBM", HistGradientBoostingClassifier())]:
print(name, round(m.fit(Xtr, ytr).score(Xte, yte), 3)) # GBM usually wins on tablesChapter summary
Linear models for smooth, additive, scarce-data signal; trees for nonlinear, interacting, messy mixed features.
Trees beat deep nets on most tables due to inductive bias: irregular targets, uninformative features, non-rotation-invariance.
Deep learning wins with huge data, embeddings, multi-modal signal, or foundation models (TabPFN) on small data.
Use the decision guide; gradient boosting is the default first move on a new table.
The real lesson is inductive bias: match the model’s assumptions to the data’s structure.