Before we multiply trees into forests and boosters, it pays to understand a single tree’s character honestly: what it does brilliantly, where it fails, and — most importantly — the bias–variance behavior that motivates every ensemble method to come. The headline: a deep tree has low bias but high variance, and that single fact explains both bagging and boosting.

What trees do well

Decision trees earn their place because they are low-maintenance and flexible:

  • No feature scaling. Splits depend only on order, so standardization and normalization are irrelevant.

  • Mixed data, minimal prep. Numeric and categorical features, different scales, outliers — handled with little fuss.

  • Nonlinearity and interactions for free. Captured by the arrangement of splits, no manual feature engineering.

  • Interpretable (when small). A shallow tree is a readable flowchart; feature importance is built in.

  • Fast to train and predict.

The core weakness: high variance

A model has high variance if small changes in the training data produce large changes in the fitted model. Deep decision trees are the textbook example: because each split is chosen greedily and all descendant splits depend on it, perturbing a few points can change the root split and reshuffle the entire tree.

Picture the bias–variance tradeoff as tree depth grows. A stump (depth 1) is stable but crude — high bias, low variance, it underfits. A fully grown tree fits the training data perfectly — low bias, but wildly high variance; the staircase chases noise. Somewhere between lies the best single tree, but its test error is still limited by that variance. The key realization: trees are unstable, low-bias learners. That is exactly the raw material averaging loves.

Other limitations

Beyond variance, single trees have real weaknesses: (1) no extrapolation (Chapter [ch:treeregression]); (2) axis-aligned only — a diagonal boundary must be approximated by a clumsy staircase of horizontal/vertical cuts; (3) biased splits toward high-cardinality features; (4) instability makes “the” tree hard to trust for inference; and (5) a tendency to create unbalanced trees on skewed data. Most of these are cured or muted by ensembling — the price is interpretability.

From one tree to many

Here is the pivot of the whole book. Averaging \(B\) independent, identically distributed predictions with variance \(\sigma^2\) yields variance \(\sigma^2/B\); if they are only correlated with correlation \(\rho\), the averaged variance is \[\rho\,\sigma^2 + \frac{1-\rho}{B}\,\sigma^2 .\] So to slash variance you want many trees (\(B\) large) that are decorrelated (\(\rho\) small). High-variance, low-bias trees are the ideal ingredient: averaging them barely touches bias but crushes variance. This single formula motivates bagging (grow many trees on resampled data) and random forests (also decorrelate them by random feature subsets) — Chapter [ch:randomforests]. Boosting attacks the problem from the other side, reducing bias by adding trees sequentially — Chapter [ch:boosting].

The bias–variance lens is the through-line from trees to all of ML. Random forests exploit “low bias, high variance, decorrelate and average”; gradient boosting exploits “start high-bias, reduce bias additively, regularize against variance.” Deep learning lives in the same coordinates — big networks are low-bias/high-variance and are tamed by data augmentation, dropout, and ensembling, the neural echoes of bagging. Understanding why one tree is unstable is understanding why every powerful model needs a variance-control strategy.

import numpy as np
from sklearn.tree import DecisionTreeRegressor
# Variance demo: refit a deep tree on bootstrap samples; predictions at x=5 vary a lot
rng = np.random.default_rng(0)
X = np.sort(rng.uniform(0,10,200)).reshape(-1,1); y = np.sin(X).ravel()+rng.normal(0,.2,200)
preds = []
for _ in range(50):
    idx = rng.integers(0, 200, 200)                  # bootstrap resample
    t = DecisionTreeRegressor(max_depth=None).fit(X[idx], y[idx])
    preds.append(t.predict([[5.0]])[0])
print("deep tree std at x=5:", round(np.std(preds), 3))   # high -> averaging will help

Chapter summary

  • Trees shine at minimal preprocessing, mixed data, automatic nonlinearity/interactions, and (when small) interpretability.

  • Their core flaw is high variance: deep trees are low-bias but unstable, reshuffling under small data changes.

  • Other limits: no extrapolation, axis-aligned (staircase) boundaries, biased splits, instability.

  • Averaging \(B\) trees with correlation \(\rho\) gives variance \(\rho\sigma^2+\frac{1-\rho}{B}\sigma^2\) — the case for many, decorrelated trees.

  • This bias–variance picture motivates bagging/forests (cut variance) and boosting (cut bias) — the rest of the book.