Integration is far more than “area under a curve.” Any quantity built by accumulating infinitely many infinitesimal pieces is an integral: areas between curves, volumes of solids, lengths of arcs, averages, and — most important for us — the probabilities and expectations of continuous random variables. This chapter shows the pattern and then turns squarely toward the probability that machine learning runs on.

Area between curves

The area between \(y=f(x)\) and \(y=g(x)\) (with \(f\ge g\)) over \([a,b]\) is \[A=\int_a^b\big[f(x)-g(x)\big]\,\dd x.\] The recipe generalizes: to measure something, slice it into infinitesimal pieces, write the size of one slice, and integrate. This “slice and sum” habit is the real content of applied integration.

Volumes

Slicing a solid into thin disks or shells gives its volume. A solid of revolution about the \(x\)-axis has \[V=\int_a^b \pi\,[f(x)]^2\,\dd x\quad(\text{disks}),\] since each slice is a disk of area \(\pi r^2=\pi f(x)^2\) and thickness \(\dd x\). Other geometries (shells, washers, cross-sections) follow the same slice-and-integrate logic.

Average value of a function

The average value of \(f\) over \([a,b]\) is \[\bar f = \frac{1}{b-a}\int_a^b f(x)\,\dd x.\]

This is the continuous analogue of averaging a list: sum (integrate) the values and divide by the size of the domain. It is the bridge to the expected value — the average of a function weighted by a probability density.

Probability densities and expectation

This is the application that matters most for AI, so we treat it carefully.

A probability density function (pdf) \(p(x)\) satisfies \(p(x)\ge0\) and \(\displaystyle\int_{-\infty}^{\infty}p(x)\,\dd x=1\). The probability that \(X\) lands in \([a,b]\) is the area \(\displaystyle\Prob(a\le X\le b)=\int_a^b p(x)\,\dd x\).

The expected value (mean) and variance of a continuous random variable are integrals: \[\E[X]=\int_{-\infty}^{\infty} x\,p(x)\,\dd x,\qquad \Var[X]=\int_{-\infty}^{\infty}(x-\E[X])^2\,p(x)\,\dd x.\] More generally \(\displaystyle\E[g(X)]=\int g(x)\,p(x)\,\dd x\) — value times probability, accumulated.

An expectation is a weighted average over a continuum: each value \(x\) contributes \(g(x)\), weighted by how likely it is, \(p(x)\,\dd x\). This single idea — “integrate value against density” — is the form of nearly every loss and objective in probabilistic machine learning. Because the integrals are usually intractable, we estimate them by sampling: \(\E[g(X)]\approx\frac1N\sum_i g(x_i)\).

Entropy, cross-entropy, and KL divergence

Information theory — the source of classification losses — is built entirely from integrals (or sums, in the discrete case) of a density against its own logarithm: \[H(p)=-\!\int p\ln p\,\dd x,\quad H(p,q)=-\!\int p\ln q\,\dd x,\quad \KL(p\,\|\,q)=\int p\ln\frac{p}{q}\,\dd x.\] Entropy \(H(p)\) measures uncertainty; cross-entropy \(H(p,q)\) measures the cost of using model \(q\) for true distribution \(p\); their difference is the KL divergence, a (non-symmetric) “distance” between distributions with \(\KL\ge0\) and \(=0\) iff \(p=q\). Minimizing cross-entropy — the standard classification loss — is minimizing KL between the data and the model.

The objects above are the loss functions of machine learning. Training a classifier minimizes cross-entropy; fitting a probabilistic model maximizes a log-likelihood, which is an expectation (integral) over the data; variational methods minimize KL divergence; the ELBO that trains VAEs is a difference of an expected log-likelihood and a KL term — all integrals (Chapter [ch:calcmlai]). Since these integrals rarely have closed forms, we replace them by averages over mini-batches and samples — the reason “expectation” and “sample mean” are used almost interchangeably in practice.

import numpy as np
# Expectation by Monte Carlo: E[X^2] for X ~ N(0,1) is 1 (= variance)
x = np.random.randn(1_000_000)
print(x.mean(), (x**2).mean())            # ~0, ~1

# KL divergence between two discrete distributions (sum form of the integral)
p = np.array([0.2, 0.5, 0.3])
q = np.array([0.1, 0.6, 0.3])
kl = np.sum(p * np.log(p / q))
print(kl)                                  # >= 0, zero only if p == q

Chapter summary

  • Applied integration is “slice into infinitesimal pieces and sum”: areas, volumes, arc length, averages.

  • The average value \(\tfrac{1}{b-a}\int_a^b f\) generalizes to the expectation \(\E[g(X)]=\int g\,p\,\dd x\).

  • A pdf integrates to \(1\); probabilities are areas; mean and variance are integrals.

  • Entropy, cross-entropy, KL are integrals of densities against logs — the basis of classification and variational losses.

  • These integrals are usually estimated by sampling, linking continuous theory to mini-batch practice.