Trees predict numbers as easily as labels. A regression tree partitions the feature space the same way, but each leaf predicts a constant value — the average target in that region — and splits are chosen to reduce variance rather than class impurity. The result is a piecewise-constant surface: a staircase that approximates any function, with characteristic strengths and one notable weakness.

Splitting to reduce variance

For a regression tree, a node’s impurity is the variance (mean squared error) of its targets around their mean \(\bar y\): \[I(\text{node}) = \frac{1}{n}\sum_{i\in\text{node}}(y_i-\bar y)^2 .\] A split is chosen to maximize the variance reduction — parent MSE minus the weighted average child MSE — exactly the information-gain formula with variance as the impurity.

The optimal constant prediction in a region, under squared error, is the mean of the targets there. So a regression tree fits a step function: the feature space is carved into boxes, and each box predicts its average \(y\). Minimizing within-node variance is the same as making each box’s points as flat (similar in \(y\)) as possible — the regression analog of making each box pure in class.

Reading a regression tree

Prediction is identical to classification: route the input down the questions to a leaf, then output that leaf’s stored average. Deeper trees have more, smaller boxes and finer steps; a tree of depth \(d\) can have up to \(2^d\) leaves. Because each region is a flat constant, a regression tree captures nonlinearity and interactions through the arrangement of cuts, not through any smooth formula.

The extrapolation weakness

A regression tree cannot extrapolate. Its prediction is always an average of training targets, so for inputs beyond the training range it just repeats the nearest leaf’s constant — a flat line, never a rising trend. Feed a tree trained on house sizes 50–200 m\(^2\) a 400 m\(^2\) mansion and it predicts the same price as a 200 m\(^2\) house. Linear models extrapolate (sometimes wrongly, but they try); trees refuse. For trending data (time series, growth), detrend first or use a model that can extrapolate.

The flip side is robustness: because predictions are bounded by observed targets and based on order (not magnitude), regression trees are insensitive to outliers in features and to monotonic feature transforms — taking logs of a predictor does not change a single split. This is part of why trees need so little preprocessing.

Regression trees are the base learners of gradient boosting — and there, crucially, the trees do not fit the target \(y\) directly but the negative gradient of a loss (for squared error, the residuals). Each boosting round adds a small regression tree that nudges predictions toward the truth (Chapter [ch:boosting]). So the humble “predict the mean in each box” rule, repeated thousands of times on residuals, becomes XGBoost — the model behind a large share of winning tabular solutions. The extrapolation weakness also explains a known failure mode of boosted trees on trending targets.

import numpy as np
from sklearn.tree import DecisionTreeRegressor
rng = np.random.default_rng(0)
X = np.sort(rng.uniform(0, 10, 120)).reshape(-1, 1)
y = np.sin(X).ravel() + rng.normal(0, 0.1, 120)
for d in [2, 5]:
    t = DecisionTreeRegressor(max_depth=d).fit(X, y)
    print(f"depth {d}: leaves={t.get_n_leaves()}, train R^2={t.score(X, y):.3f}")
print("extrapolation at x=20:", t.predict([[20]]).round(3),  # flat, = nearest leaf mean
      " vs at x=10:", t.predict([[10]]).round(3))

Chapter summary

  • A regression tree predicts the mean target in each leaf region; splits maximize variance reduction (MSE).

  • The model is piecewise-constant — a staircase that captures nonlinearity and interactions via the arrangement of cuts.

  • Trees cannot extrapolate: predictions are bounded by training targets and flatten outside the training range.

  • They are robust to outliers and invariant to monotonic feature transforms — hence minimal preprocessing.

  • Regression trees on residuals/gradients are the base learners of gradient boosting (XGBoost, LightGBM, CatBoost).