A handful of distributions describe most of the randomness we ever model. Each arises from a simple, recurring story — a coin flip, a count of rare events, a sum of many small effects — and learning those stories lets you recognize at a glance which distribution fits a problem. This chapter is your field guide to the essential discrete and continuous distributions, with special attention to the Gaussian, which dominates statistics and machine learning.

Discrete distributions

\(X\in\{0,1\}\) with \(P(X=1)=p\). Mean \(p\), variance \(p(1-p)\). The atom of binary classification: a single labeled example is a Bernoulli outcome.

\(P(X=k)=\binom{n}{k}p^k(1-p)^{n-k}\), mean \(np\), variance \(np(1-p)\). Counts of correct predictions, conversions, defects.

Models waiting times in discrete steps; “memoryless.”

\(P(X=k)=\dfrac{\lambda^k e^{-\lambda}}{k!}\), mean \(=\) variance \(=\lambda\). Arrivals, clicks, mutations, word counts. It is the limit of Binomial when \(n\to\infty\), \(p\to0\) with \(np=\lambda\) fixed.

The multi-class generalization of Bernoulli/Binomial; the target distribution of every softmax classifier.

Continuous distributions

Density \(\tfrac{1}{b-a}\). The basis of random number generation and uninformative priors.

Density \(\lambda e^{-\lambda x}\), mean \(1/\lambda\). Memoryless; the continuous cousin of the geometric, paired with the Poisson process.

Density \(\dfrac{1}{\sigma\sqrt{2\pi}}\exp\!\left(-\dfrac{(x-\mu)^2}{2\sigma^2}\right)\). The most important distribution in all of statistics (next section).

The natural prior for a Bernoulli/Binomial rate; conjugate (Chapter [ch:bayesian]).

The \(t\) has heavy tails and governs tests when \(\sigma\) is unknown (Chapter [ch:testing]).

The Gaussian and why it is everywhere

The normal distribution earns its dominance for three reasons.

(1) The Central Limit Theorem (Chapter [ch:limit]): sums and averages of many independent effects are approximately Gaussian, regardless of the original distribution. Since measurement noise, aggregate behavior, and averaged gradients are all such sums, Gaussians appear by default. (2) Maximum entropy: among all distributions with a given mean and variance, the Gaussian is the most “spread out” / least presumptuous — the natural choice when you know only the first two moments. (3) Mathematical convenience: it is closed under linear maps, conditioning, and marginalization, and its log is a simple quadratic — which is exactly why squared-error loss corresponds to a Gaussian noise model.

About \(68\%\) of a Gaussian’s mass lies within one standard deviation of the mean, \(95\%\) within two, and \(99.7\%\) within three — the rule of thumb behind “\(2\sigma\)” error bars and many confidence intervals.

Relationships between distributions

These distributions form a connected family, not a list to memorize: Bernoulli trials sum to a Binomial; the Binomial tends to a Poisson (rare events) and to a Gaussian (many events, by the CLT); the Exponential is the continuous Geometric; the Beta is a prior for the Bernoulli rate; sums of squared Gaussians are \(\chi^2\). Seeing the web of relationships is what turns a catalog into understanding.

Choosing a distribution is choosing a model, and that choice fixes the loss function (Chapter [ch:estimation]). Modeling labels as Categorical and maximizing likelihood yields cross-entropy loss — the default for classification. Modeling targets as Gaussian yields mean-squared-error loss — the default for regression. Modeling counts as Poisson yields Poisson regression. The Beta and Gaussian serve as conjugate priors for Bayesian models; the Bernoulli underlies dropout; the Gaussian is the latent prior in VAEs and the noise model in diffusion. When you pick a distribution, you are telling the model what kind of randomness generated the data.

import numpy as np
from scipy import stats
# Binomial -> Gaussian (CLT) and Binomial -> Poisson (rare events)
print(stats.binom.pmf(4, 10, 0.4))          # ~0.2508
print(stats.poisson.pmf(3, 3))              # ~0.2240
# 68-95-99.7 rule for the standard normal
for k in (1, 2, 3):
    print(k, round(stats.norm.cdf(k) - stats.norm.cdf(-k), 4))   # .6827 .9545 .9973

Chapter summary

  • Each distribution encodes a story: Bernoulli (one trial), Binomial (count of successes), Poisson (rare events), Exponential (waiting time), Gaussian (sums/averages).

  • The Gaussian dominates via the CLT, maximum entropy, and analytic convenience; remember the 68–95–99.7 rule.

  • Distributions are related (Bernoulli\(\to\)Binomial\(\to\)Poisson/Gaussian; Beta as a rate prior), not isolated.

  • Choosing a distribution chooses the loss: Categorical \(\to\) cross-entropy, Gaussian \(\to\) MSE, Poisson \(\to\) Poisson regression.