So far events have been qualitative (“it rained”). A random variable attaches numbers to outcomes, letting us do arithmetic with uncertainty: average it, measure its spread, add independent copies. Random variables and their two summary numbers — expectation (the center) and variance (the spread) — are the daily vocabulary of statistics and machine learning.

Random variables

A random variable \(X\) is a function from the sample space to the real numbers, \(X:\Omega\to\R\). It is discrete if it takes countably many values (a die, a word count) and continuous if it takes values in a continuum (a height, a measurement).

A random variable is described by its distribution, which says how probability is spread over its values:

  • Discrete: the probability mass function (PMF) \(p(x)=P(X=x)\), with \(p(x)\ge0\) and \(\sum_x p(x)=1\).

  • Continuous: the probability density function (PDF) \(f(x)\ge0\) with \(\int f(x)\,\dd x=1\); probabilities are areas, \(P(a\le X\le b)=\int_a^b f(x)\,\dd x\).

  • Either: the cumulative distribution function (CDF) \(F(x)=P(X\le x)\), non-decreasing from \(0\) to \(1\).

For a continuous variable, \(f(x)\) is a density, not a probability — it can exceed \(1\), and \(P(X=x)=0\) for any single point. Only areas (integrals) are probabilities. Confusing density with probability is the classic continuous-variable error; it is why we never write “\(P(X=x)\)” for a continuous \(X\) but instead work with \(f(x)\,\dd x\) over a region.

Expectation

The expected value (mean) of \(X\) is its probability-weighted average: \[\E[X]=\sum_x x\,p(x)\quad(\text{discrete}),\qquad \E[X]=\int x\,f(x)\,\dd x\quad(\text{continuous}).\]

The expectation is the balance point of the distribution — where it would balance if the probability were physical weight on a ruler. It is the long-run average value over many repetitions (a fact the law of large numbers will make precise in Chapter [ch:limit]). It need not be a value \(X\) can actually take: the expected roll of a die is \(3.5\).

For any random variables and constants, \[\E[aX+bY+c]=a\,\E[X]+b\,\E[Y]+c.\] This holds even when \(X\) and \(Y\) are dependent — a remarkably powerful fact.

The law of the unconscious statistician (LOTUS) lets us take expectations of functions without finding their distribution: \(\E[g(X)]=\sum_x g(x)p(x)\) or \(\int g(x)f(x)\,\dd x\).

Variance and standard deviation

The variance measures spread around the mean: \[\Var[X]=\E\!\big[(X-\E[X])^2\big]=\E[X^2]-(\E[X])^2\ \ge 0.\] Its square root \(\sigma=\sqrt{\Var[X]}\) is the standard deviation, in the same units as \(X\).

Key scaling rules: \(\Var[aX+b]=a^2\Var[X]\) (shifts don’t change spread; scaling squares it), and for independent \(X,Y\), \(\Var[X+Y]=\Var[X]+\Var[Y]\).

Higher moments and standardization

The mean and variance are the first two moments. Higher moments describe shape: skewness (asymmetry) and kurtosis (tail heaviness). Standardizing a variable, \[Z=\frac{X-\mu}{\sigma},\] gives mean \(0\) and variance \(1\) — exactly the operation behind feature normalization and batch/layer normalization in neural networks, which keeps activations on a stable scale.

Random variables are the data of machine learning, and expectation is the form of nearly every objective. A loss is a random variable depending on a random data point; we minimize its expectation, the risk \(\E[\text{loss}]\), estimated by the average loss over a mini-batch (Chapter [ch:limit]). Linearity of expectation justifies summing per-example gradients. Variance is the language of the bias–variance tradeoff (Chapter [ch:regression]), of gradient-estimate noise in SGD, and of uncertainty in predictions. Standardization (\(Z\)-scoring) is the preprocessing and normalization that makes deep nets trainable. Even a model’s output is a random variable: a distribution we summarize by its mean (the prediction) and variance (the confidence).

import numpy as np
# A fair die: PMF, mean (3.5), variance (35/12 ~ 2.917)
x = np.arange(1, 7); p = np.ones(6)/6
mean = (x*p).sum()
var  = ((x**2)*p).sum() - mean**2
print(mean, round(var, 3))            # 3.5  2.917

# Linearity & standardization on samples
s = np.random.normal(loc=5, scale=2, size=100_000)
z = (s - s.mean())/s.std()
print(round(s.mean(),2), round(s.var(),2), round(z.mean(),2), round(z.std(),2))  # ~5, ~4, ~0, ~1

Chapter summary

  • A random variable maps outcomes to numbers; describe it by PMF (discrete), PDF (continuous), or CDF.

  • A continuous density is not a probability — only areas (integrals) are; \(P(X=x)=0\) pointwise.

  • Expectation \(\E[X]\) is the probability-weighted average (balance point); it is linear even for dependent variables.

  • Variance \(\E[X^2]-\E[X]^2\) measures spread; \(\Var[aX+b]=a^2\Var[X]\), and variances add for independent sums.

  • Standardizing \(Z=(X-\mu)/\sigma\) underlies feature and batch normalization; risk = expected loss is the ML objective.