Preface: how to use this book
There is a quiet truth in applied machine learning: while deep networks dominate images, audio, and text, the workhorse for the tables that run businesses — spreadsheets of customers, transactions, sensors, and records — is a forest of decision trees. Gradient-boosted trees win the majority of tabular competitions and power countless production systems precisely because they fit the messy, irregular structure of real-world tabular data better than almost anything else. This book builds tree-based methods from a single “yes/no” question all the way to the libraries (XGBoost, LightGBM, CatBoost) behind that dominance.
Who this is for
This book starts from the very beginning — what it means to split data with a question — and climbs to ensembles, gradient boosting, model interpretation, and the research frontier where trees meet deep learning. It is for the self-learner who wants both fluency (“I can train, tune, and interpret a gradient-boosting model and trust it”) and understanding (“I know why a single tree overfits, why averaging and boosting fix it, and why trees beat neural nets on tables”). Every major idea gets a picture, a worked example, and a forward link to machine learning.
The central idea
All of tree-based learning grows from one move and its consequences:
Partition, then predict. A tree recursively splits the feature space into boxes with simple yes/no questions, then predicts a constant in each box — the conditional mean or majority class.
One tree is weak; many are strong. A single deep tree overfits (high variance). Averaging independent trees (bagging, random forests) cuts variance; adding corrective trees (boosting) cuts bias.
Trees fit tabular reality. Axis-aligned splits handle mixed types, missing values, outliers, and irregular, non-smooth relationships with almost no preprocessing.
The structure
Part I — Foundations (beginner): what tree-based methods are; classification trees; regression trees.
Part II — Building Trees Well (core): growing, splitting, and pruning; the strengths, weaknesses, and bias–variance of trees.
Part III — Ensembles (advanced): bagging and random forests; boosting and gradient boosting; the modern libraries.
Part IV — Interpretation & Practice (advanced/expert): feature importance and SHAP; practical modeling end to end.
Part V — Frontier: specialized trees; trees versus the rest of ML; and a capstone on trees in ML/DL/AI.
How to read it
Read with a pen and a computer. When you meet a Definition, restate it; when you meet an Example, work it before reading on; when you meet a Try it in code box, run it. The colored boxes recur throughout:
Key idea — the one sentence to remember.
Intuition — the picture or analogy behind the math.
Common pitfall — the mistakes that bite everyone.
How this powers ML / AI — the forward link to applications.
Try it in code — a hands-on check in Python (scikit-learn / XGBoost).
Hold one picture above all others: a tree carves the feature space into rectangles with simple questions and predicts a constant in each. A single tree is interpretable but unstable; the power comes from combining many trees. Bagging and random forests average trees to kill variance; boosting stacks trees to kill bias. Master the single tree, and forests, XGBoost, and the tabular state of the art are the same idea, scaled.