Beyond classification, regression, forests, and boosting lies a rich ecosystem of tree variants tuned for special jobs: detecting anomalies, ranking search results, modeling time-to-event, quantifying uncertainty, and squeezing the last drop of accuracy by combining models. This chapter surveys the most useful advanced trees and ensembling tricks an expert reaches for.

Extremely randomized trees (ExtraTrees)

ExtraTrees push randomization further than random forests: instead of searching for the best threshold on each candidate feature, they pick split thresholds at random and keep the best among those. This adds bias slightly but cuts variance and training time, sometimes beating random forests — and it uses the whole dataset (no bootstrap) by default. A cheap, strong alternative worth trying alongside RF.

Isolation Forests: anomaly detection

An Isolation Forest detects outliers by exploiting a simple asymmetry: anomalies are “few and different,” so random splits isolate them in fewer steps than normal points. Build trees with random feature/threshold splits; the average path length to isolate a point becomes its anomaly score (short path \(=\) anomalous). It is unsupervised, linear-time, and scales to high dimensions — a workhorse for fraud, intrusion, and defect detection.

Gradient-boosted trees for ranking

Search engines and recommenders need to order items, not score them in isolation. Learning to rank with boosted trees (LambdaMART — the basis of much of web search ranking) optimizes ranking metrics (NDCG, MAP) by defining gradients (“lambdas”) from pairwise preferences. XGBoost, LightGBM, and CatBoost all ship rank objectives. Tree ensembles remain dominant in production ranking because of speed, accuracy, and feature flexibility.

Survival, quantile, and probabilistic trees

Trees flex to many output types by changing the loss: survival trees / random survival forests model time-to-event with censoring (medicine, churn, reliability); quantile regression forests and quantile-loss boosting predict intervals (e.g. 5th–95th percentile) rather than a point, giving calibrated uncertainty; and NGBoost (natural gradient boosting) predicts full probability distributions. These bring uncertainty quantification — usually a deep-learning selling point — to tabular trees.

Bayesian and oblique trees

BART (Bayesian Additive Regression Trees) is a sum-of-trees model with priors that keep each tree weak, fit by MCMC — giving full posterior uncertainty and strong performance with little tuning. Oblique trees split on linear combinations of features (\(a^\top x < t\)) rather than single axes, capturing diagonal boundaries that standard trees approximate with staircases — at the cost of interpretability. Explainable Boosting Machines (EBM) are glass-box GA\(^2\)Ms: boosted trees restricted to one/two features at a time, yielding accuracy near full GBMs with full additive interpretability.

Stacking and blending

Stacking trains a meta-learner on the out-of-fold predictions of several base models (e.g. a forest, an XGBoost, a neural net), letting it learn how to best combine them; blending is the simpler held-out-set version. Diverse base learners — especially trees plus a neural net — often stack into the best tabular models, and stacking is the staple of winning Kaggle solutions. The cost is complexity and inference latency.

Advanced does not mean better by default. ExtraTrees/oblique/BART can underperform a well-tuned XGBoost while costing more. Stacking shines only with diverse, decorrelated base models and rigorous out-of-fold discipline — naive stacking leaks and overfits. And every added layer trades away the interpretability and operational simplicity that made trees attractive. Reach for these when a measured gain justifies the complexity, not reflexively.

This menagerie shows trees adapting to jobs once thought to need bespoke models: Isolation Forests for unsupervised anomaly detection, LambdaMART for the ranking systems behind search and recommendation, quantile/NGBoost/BART for uncertainty quantification (the tabular answer to Bayesian deep learning), and stacking that fuses trees with neural networks. The recurring theme of the next chapters: trees are not a single algorithm but a flexible substrate that keeps absorbing tasks across machine learning and AI.

from sklearn.ensemble import IsolationForest
import numpy as np
rng = np.random.default_rng(0)
X = np.r_[rng.normal(0, 0.4, (200, 2)), rng.uniform(-4, 4, (12, 2))]  # cluster + outliers
iso = IsolationForest(contamination=0.05, random_state=0).fit(X)
scores = iso.score_samples(X)                  # lower = more anomalous
print("flagged anomalies:", int((iso.predict(X) == -1).sum()))

Chapter summary

  • ExtraTrees randomize thresholds for lower variance; Isolation Forests score anomalies by isolation path length.

  • Boosted trees handle ranking (LambdaMART/NDCG), survival, quantile, and distributional (NGBoost) outputs.

  • BART (Bayesian), oblique (linear splits), and EBM (glass-box additive) extend the tree idea in different directions.

  • Stacking/blending fuse diverse base models (trees + nets) for top accuracy — with leakage risk and added complexity.

  • Advanced variants pay off selectively; weigh the gain against lost interpretability and simplicity.