Probability is the mathematics of uncertainty — a precise language for reasoning about things we cannot predict with certainty. A coin flip, tomorrow’s weather, whether an email is spam, what word comes next in a sentence: all are uncertain, and probability lets us quantify that uncertainty and compute with it. This chapter lays the foundation: the vocabulary of outcomes and events, the three axioms that govern all of probability, the art of counting, and the two great schools of thought about what a probability means.
Sample spaces, outcomes, and events
A random experiment is any process with an uncertain outcome. The sample space \(\Omega\) is the set of all possible outcomes. An event \(A\) is a subset of \(\Omega\) — a collection of outcomes we care about.
Roll a six-sided die. The sample space is \(\Omega=\{1,2,3,4,5,6\}\). The event “even” is \(A=\{2,4,6\}\); the event “greater than 4” is \(B=\{5,6\}\). Events combine with set operations: \(A\cup B\) (“\(A\) or \(B\)”), \(A\cap B\) (“\(A\) and \(B\)”), and \(A^c\) (“not \(A\)”).
Think of \(\Omega\) as the space of everything that could happen, and an event as a region carved out of it. Asking “what is the probability of \(A\)?” is asking “how much of the total possibility lands in region \(A\)?” All the logic of probability is the logic of sets — unions, intersections, complements — with a number attached that measures size.
The axioms of probability
Modern probability rests on three rules, due to Kolmogorov (1933). Everything else is derived from them.
A probability \(P\) assigns to each event a number such that
Nonnegativity: \(P(A)\ge 0\) for every event \(A\).
Normalization: \(P(\Omega)=1\) — something must happen.
Additivity: if \(A_1,A_2,\dots\) are mutually exclusive (disjoint), then \(P\!\left(\bigcup_i A_i\right)=\sum_i P(A_i)\).
From these follow all the familiar rules: \[P(A^c)=1-P(A),\qquad P(\varnothing)=0,\qquad P(A\cup B)=P(A)+P(B)-P(A\cap B).\]
The last rule — inclusion–exclusion — is where beginners stumble: you may only add probabilities directly when events are disjoint. For overlapping events you must subtract the double-counted intersection. Forgetting this is the single most common probability error, and it reappears in subtle forms (e.g. assuming features are independent when they are not).
Equally likely outcomes and counting
When a finite sample space has \(N\) equally likely outcomes, probability reduces to counting: \[P(A)=\frac{\text{number of outcomes in }A}{\text{total number of outcomes}}=\frac{|A|}{|\Omega|}.\] So we need to count well. The two workhorses are permutations (ordered arrangements) and combinations (unordered selections): \[\text{permutations: } \frac{n!}{(n-k)!},\qquad \text{combinations: } \binom{n}{k}=\frac{n!}{k!\,(n-k)!}.\]
A standard deck has \(\binom{52}{5}=2{,}598{,}960\) possible five-card poker hands. The probability of a specific hand is one in that many. Counting the hands of a given type (e.g. a flush) and dividing gives its probability.
Combinations \(\binom{n}{k}\) count “how many ways to choose \(k\) things from \(n\) when order doesn’t matter.” They appear far beyond card games: the binomial distribution (Chapter [ch:distributions]) counts the ways a fixed number of successes can occur, and the same coefficients show up whenever we sum over subsets — including the combinatorial terms in model-counting and ensemble methods.
What does a probability mean? Two interpretations
The axioms tell us how probabilities behave, but not what they mean. Two schools answer differently — and the split runs straight through machine learning.
Frequentist: a probability is a long-run frequency. “\(P(\text{heads})=0.5\)” means that in many repeated flips, the fraction of heads approaches \(0.5\). Probability is a property of the world, revealed by repetition.
Bayesian: a probability is a degree of belief. “\(P(\text{rain tomorrow})=0.3\)” is a coherent statement of confidence given current information — even though tomorrow happens only once. Probability is a property of the observer’s knowledge, updated as evidence arrives.
Neither view is “correct”; they are different tools. Frequentist reasoning underlies hypothesis tests and confidence intervals (Chapter [ch:testing]); Bayesian reasoning underlies posterior updating and most modern probabilistic ML (Chapter [ch:bayesian]). Deep learning quietly uses both: it estimates parameters by (frequentist) maximum likelihood, yet interprets a softmax output as a (Bayesian-flavored) degree of belief over classes. You will become fluent in both dialects.
Every prediction a model makes is a probability. A classifier does not output “cat”; it outputs a distribution like \(P(\text{cat})=0.83,\ P(\text{dog})=0.12,\dots\) obeying exactly the axioms above (nonnegative, summing to one). The sample space is the set of classes; the softmax layer is a machine for producing a valid probability over it. Generative models go further and place a probability over all possible images or sentences. Getting the foundations right — what is an event, when may probabilities be added, what does the number mean — is what separates a model that reasons soundly about uncertainty from one that fools itself.
import numpy as np
from math import comb
# Frequentist view: long-run frequency approaches the probability
flips = np.random.randint(0, 2, size=1_000_000) # 0/1 = tails/heads
print(flips.mean()) # -> ~0.5
# Counting: probability of being dealt a flush-shaped 5-card combo
total = comb(52, 5)
print(total) # 2,598,960
# inclusion-exclusion: P(A or B) with overlap
PA, PB, PAB = 0.5, 0.4, 0.2
print(PA + PB - PAB) # 0.7, NOT 0.9Chapter summary
The sample space \(\Omega\) holds all outcomes; an event is a subset; probability measures the “size” of events.
Kolmogorov’s axioms (nonnegativity, normalization, additivity for disjoint events) generate all the rules.
Add probabilities only for disjoint events; otherwise use \(P(A\cup B)=P(A)+P(B)-P(A\cap B)\).
For equally likely outcomes, probability is counting: permutations (ordered) and combinations \(\binom nk\) (unordered).
Frequentist (long-run frequency) and Bayesian (degree of belief) interpretations both run through machine learning.