A single tree is a readable flowchart, but a forest of hundreds or a thousand boosted trees is a black box. The good news: tree ensembles support some of the best interpretability tools in machine learning. This chapter covers built-in (impurity) importance and its biases, the more trustworthy permutation importance, the gold-standard SHAP values, and partial-dependence plots — enough to explain a model to a stakeholder and to catch leakage.

Built-in (impurity / gain) importance

Impurity importance (Mean Decrease in Impurity, MDI) scores a feature by the total impurity reduction (Gini/variance, or split gain in boosting) it contributes, summed over all splits that use it and averaged over all trees.

Impurity importance is fast and free but biased: it inflates the importance of high-cardinality features (many split points \(\Rightarrow\) more chances to reduce impurity, even by chance) and continuous features over binary ones, and it is computed on training data so it rewards overfitting. With correlated features, importance is split arbitrarily among them. Treat MDI as a rough first look, never as ground truth.

Permutation importance

Permutation importance measures how much a model’s validation score drops when a feature’s values are randomly shuffled, breaking its relationship with the target. It is model-agnostic, computed on held-out data, and directly answers “how much does the model rely on this feature for accuracy?” — avoiding MDI’s cardinality bias. Its caveat: with correlated features, shuffling one is compensated by its correlate, so both can look unimportant; group correlated features or use conditional variants.

SHAP: consistent, local attributions

SHAP (SHapley Additive exPlanations) assigns each feature a contribution to each individual prediction, grounded in cooperative game theory: a feature’s SHAP value is its average marginal contribution over all orderings of features. SHAP is locally accurate (per-prediction contributions sum to the prediction minus the baseline) and consistent (if a model relies on a feature more, its SHAP value cannot decrease). TreeSHAP computes exact Shapley values for tree ensembles in polynomial time — making SHAP the modern standard for explaining tabular models both locally (one prediction) and globally (aggregated).

Read SHAP two ways. A waterfall/force plot explains one prediction: start at the baseline (average output), then each feature pushes the prediction up or down by its SHAP value until you reach the final score — “this loan was denied mainly because of high debt (\(+0.3\)) despite good income (\(-0.1\)).” A summary (beeswarm) plot aggregates across all samples: features ranked by importance, with color showing whether high or low feature values push predictions up or down — revealing direction and interactions, not just magnitude.

Partial dependence and ICE

A partial dependence plot (PDP) shows the average predicted output as one feature varies, marginalizing the others — the shape of the learned relationship (monotone? threshold? U-shaped?). Individual Conditional Expectation (ICE) curves show one line per sample, exposing heterogeneity and interactions that a single averaged PDP hides. PDPs assume feature independence, so they mislead when features are strongly correlated (they evaluate unrealistic combinations).

Interpretation pitfalls: (1) importance \(\ne\) causation — a feature can be important because it proxies a confounder or leaks the target; (2) a single feature with implausibly dominant importance/SHAP is a classic leakage red flag (e.g., an ID, a post-outcome timestamp); (3) PDP/permutation both distort under correlation; (4) explanations describe the model, not the world — a wrong model yields confident, wrong explanations. Use these tools to debug and communicate, not to prove mechanisms.

SHAP began with trees but became a unifying language for explainable AI across all models, including deep networks (DeepSHAP, GradientSHAP) — the same Shapley-value foundation. Tree-based importances and SHAP are everywhere in production ML: detecting data leakage, satisfying regulatory “right to explanation” (credit, insurance), monitoring feature drift, and selecting features. Because TreeSHAP is exact and fast, gradient-boosted trees are often the most auditable high-accuracy models available — a major reason regulated industries prefer them over neural nets (Chapter [ch:treemlai]).

from sklearn.ensemble import RandomForestClassifier
from sklearn.inspection import permutation_importance
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
X, y = load_breast_cancer(return_X_y=True)
Xtr, Xte, ytr, yte = train_test_split(X, y, random_state=0)
rf = RandomForestClassifier(n_estimators=300, random_state=0).fit(Xtr, ytr)
r = permutation_importance(rf, Xte, yte, n_repeats=20, random_state=0)
import numpy as np
for i in np.argsort(r.importances_mean)[::-1][:5]:
    print(f"feat {i}: {r.importances_mean[i]:.3f} +/- {r.importances_std[i]:.3f}")
# import shap; shap.TreeExplainer(rf).shap_values(Xte)  # exact per-prediction attributions

Chapter summary

  • Impurity (MDI) importance is free but biased toward high-cardinality/continuous features and computed on training data.

  • Permutation importance (validation score drop under shuffling) is model-agnostic and more trustworthy — mind correlated features.

  • SHAP gives consistent, locally accurate per-prediction attributions; TreeSHAP is exact and fast for tree ensembles.

  • PDP/ICE reveal the shape of learned relationships but distort under feature correlation.

  • Importance \(\ne\) causation; a dominant feature often signals leakage. Trees + SHAP make ensembles highly auditable.