Real problems have many interacting variables. Bayesian networks (directed graphical models) make such joint distributions tractable by drawing the dependencies as a graph: an arrow means “directly influences,” a missing arrow means “conditionally independent.” The graph is both a picture of your assumptions and a computational engine for answering probabilistic queries. Naive Bayes was the simplest example; this chapter is the general theory.
Factorizing a joint distribution
A Bayesian network is a directed acyclic graph (DAG) whose nodes are random variables. It encodes the joint distribution as a product of local conditionals, each variable given its parents: \[P(X_1,\dots,X_n)=\prod_{i=1}^n P\big(X_i\mid \text{parents}(X_i)\big).\]
This factorization is the whole payoff. A joint over \(n\) binary variables needs \(2^n-1\) numbers; a Bayesian network needs only the small conditional tables for each node given its (few) parents. By asserting that most variables do not directly depend on most others, the graph turns an exponential object into a compact, interpretable one — and makes inference feasible.
Conditional independence and d-separation
The graph’s missing edges encode conditional independencies, read off by d-separation. Three canonical structures:
Chain \(A\to B\to C\): \(A\) and \(C\) are independent given \(B\) (\(B\) blocks the path).
Fork \(A\leftarrow B\to C\): a common cause; \(A,C\) dependent but independent given \(B\).
Collider \(A\to B\leftarrow C\): \(A,C\) independent, but conditioning on \(B\) (or its descendants) creates dependence — “explaining away.”
“Explaining away” is the subtle, important one: if wet grass can be caused by rain or the sprinkler, learning it rained lowers the probability the sprinkler was on, even though rain and sprinkler are independent a priori. Observing a shared effect couples its causes. This is why conditioning on the wrong variable (a collider) can create spurious correlations — the graphical version of the causality pitfalls in the regression literature.
Inference on the graph
Inference means computing a query like \(P(\text{Rain}\mid \text{Slip}=\text{true})\) by summing the factorized joint over the unobserved variables. Exact methods (variable elimination, belief propagation, the junction-tree algorithm) exploit the graph structure; when the graph is too complex, we fall back on the approximate methods of Chapter [ch:mcmc].
Inference flows both directions along the arrows. Arrows point from cause to effect (the generative story), but Bayes’ theorem lets us reason from observed effects back to hidden causes — diagnosis. A Bayesian network thus answers predictive queries (causes\(\to\)effects) and diagnostic queries (effects\(\to\)causes) within one coherent model. This bidirectional reasoning is what makes graphical models so powerful for expert systems and probabilistic AI.
The wider family
Bayesian networks are one branch of probabilistic graphical models. Undirected Markov random fields model symmetric relationships (pixels in an image, spins in a lattice); hidden Markov models and Kalman filters are dynamic graphical models for sequences; factor graphs unify them for message-passing inference. All share the core idea: structure the joint, then compute with it.
Two recurring mistakes. (1) Arrows are not automatically causal. A Bayesian network encodes conditional independence; reading edges as cause-and-effect requires extra assumptions (interventions, no hidden confounders) — that is the leap to causal graphical models. (2) Exact inference is NP-hard in general; densely connected graphs blow up, so know when to switch to approximate inference. And learning structure from data is hard — many graphs fit equally well.
Graphical models are a foundational language for probabilistic AI. Hidden Markov models drove speech recognition and bioinformatics for decades; Kalman filters (a Gaussian graphical model) power navigation, tracking, and control. Conditional random fields were the workhorse of sequence labeling before neural nets. Modern latent-variable models — VAEs, diffusion models, topic models (LDA) — are graphical models whose conditionals are parameterized by neural networks, trained with the variational inference of the next chapter. The factorization idea also underlies probabilistic programming languages (Stan, PyMC, Pyro) that let you specify a graph and infer automatically.
# Sprinkler network: P(Rain | Slip) by enumerating the factorized joint
import itertools
P_R = {1: 0.2, 0: 0.8}
P_S = {1: 0.4, 0: 0.6}
P_W = {(1,1):0.99,(1,0):0.9,(0,1):0.9,(0,0):0.0} # P(Wet=1 | R, S)
P_Sl = {1: 0.8, 0: 0.05} # P(Slip=1 | Wet)
num = den = 0.0
for r, s, w in itertools.product([0,1],[0,1],[0,1]):
pw = P_W[(r,s)] if w else 1-P_W[(r,s)]
joint = P_R[r]*P_S[s]*pw*P_Sl[w] # P(R,S,W,Slip=1)
den += joint; num += joint if r==1 else 0
print("P(Rain | Slip=1) =", round(num/den, 3))Chapter summary
A Bayesian network (DAG) factorizes a joint as \(\prod_i P(X_i\mid \text{parents})\), shrinking an exponential object to compact local tables.
Missing edges encode conditional independence, read via d-separation: chains and forks block, colliders couple (“explaining away”).
Inference sums the factorized joint over hidden variables and flows both ways: prediction (causes\(\to\)effects) and diagnosis (effects\(\to\)causes).
The family includes MRFs, HMMs, Kalman filters, and factor graphs.
Arrows are not automatically causal; exact inference is NP-hard in general. Graphical models underlie HMMs, CRFs, VAEs, diffusion, and probabilistic programming.