A decision tree is the most human of machine-learning models: it is a flowchart of yes/no questions, exactly how a doctor triages a patient or a loan officer screens an application. This chapter introduces the core idea — recursive partitioning — and the vocabulary of nodes, splits, and leaves, then previews how stacking and averaging many trees produces the methods that dominate tabular machine learning.

The flowchart model

A decision tree predicts an output by routing an input down a series of tests. Each internal node asks a yes/no question about one feature (“is age \(<45\)?”); each branch is an answer; each leaf holds a prediction — a class label (classification) or a number (regression). The path from root to leaf is a chain of conditions that defines a region of feature space.

Recursive partitioning

A tree recursively partitions the feature space into axis-aligned rectangles (“boxes”), then predicts a constant within each box — the majority class or the average target. Growing a tree means repeatedly asking: which single yes/no question splits this region into the two purest possible sub-regions? Each split is a straight cut perpendicular to one feature axis; stacking cuts builds an increasingly fine staircase that approximates the true relationship.

The tree and the partition are two views of the same object. Every leaf corresponds to one rectangle; every split corresponds to one cut. This is why trees handle interactions naturally: a split on \(x_2\) that occurs only inside the “\(x_1<2.4\)” branch means the effect of \(x_2\) depends on \(x_1\) — something a linear model must be told about explicitly, but a tree discovers automatically.

A short history

  • ID3 (1986) and C4.5 (1993), by Quinlan, grew trees using information gain (entropy); C4.5 added pruning and continuous features.

  • CART (1984), by Breiman, Friedman, Olshen, and Stone, unified classification and regression with binary splits, the Gini criterion, and cost-complexity pruning. This is what scikit-learn, XGBoost, and LightGBM implement — when practitioners say “decision tree,” they almost always mean CART.

  • Ensembles: bagging (1996) and random forests (2001) by Breiman; AdaBoost (1997) by Freund and Schapire; gradient boosting (2001) by Friedman; and the modern libraries XGBoost (2016), LightGBM (2017), and CatBoost (2018).

Why trees matter

A single tree is interpretable but unstable and weak — a small data change can reshuffle the whole tree, and a shallow tree underfits while a deep one overfits. The breakthrough is ensembling: combine many trees so their errors cancel. Bagging and random forests average many independent trees to cut variance; boosting adds trees sequentially, each fixing the last one’s mistakes, to cut bias. Ensembles of trees are, for tabular data, the most reliably accurate models we have.

Tree ensembles are the quiet champions of tabular machine learning — the spreadsheets and databases that run finance, healthcare, advertising, fraud detection, and recommendation. Gradient-boosted decision trees (XGBoost, LightGBM, CatBoost) win the majority of tabular Kaggle competitions and remain the pragmatic default in industry, often outperforming deep neural networks on structured data while training far faster (Chapter [ch:vsworld], [ch:treemlai]). They need almost no preprocessing — no scaling, native handling of mixed types and missing values — which makes them the first model most practitioners reach for on a new table.

from sklearn.tree import DecisionTreeClassifier, export_text
from sklearn.datasets import load_iris
X, y = load_iris(return_X_y=True)
tree = DecisionTreeClassifier(max_depth=2).fit(X, y)
print(export_text(tree, feature_names=load_iris().feature_names))  # read the flowchart
print("accuracy:", round(tree.score(X, y), 3))

Chapter summary

  • A decision tree is a flowchart of yes/no feature tests; root-to-leaf paths define regions, and each leaf predicts a constant.

  • Trees work by recursive partitioning into axis-aligned boxes; splits capture feature interactions automatically.

  • The dominant algorithm is CART (binary splits, Gini/variance, pruning); ID3/C4.5 are the entropy-based ancestors.

  • A single tree is interpretable but unstable; the power comes from ensembles — bagging/forests (variance) and boosting (bias).

  • Tree ensembles dominate tabular ML, with minimal preprocessing — the workhorse behind much of applied data science.