Differentiation is mechanical; integration is an art. There is no universal algorithm, so we build a toolkit of techniques — each one reversing a differentiation rule — plus numerical methods for the (many) integrals with no closed form, and improper integrals for infinite regions, which is exactly the setting of probability densities.

Substitution: reversing the chain rule

If \(u=g(x)\) then \(\dd u=g'(x)\dd x\), and \[\int f(g(x))\,g'(x)\,\dd x = \int f(u)\,\dd u.\]

Substitution is the chain rule read backwards: spot an “inside function” \(g(x)\) whose derivative also appears, and rename it \(u\) to collapse the integral into a basic one. For definite integrals, remember to convert the limits to \(u\)-values too. This is the single most-used technique.

\(\displaystyle\int 2x\,e^{x^2}\,\dd x\): let \(u=x^2\), \(\dd u=2x\,\dd x\), giving \(\int e^u\,\dd u=e^u+C=e^{x^2}+C.\)

Integration by parts: reversing the product rule

\[\int u\,\dd v = uv - \int v\,\dd u.\]

Choose \(u\) to be the part that simplifies when differentiated, and \(\dd v\) the part you can integrate. The mnemonic LIATE (Logs, Inverse-trig, Algebraic, Trig, Exponential) suggests a good \(u\).

\(\displaystyle\int x\,e^x\,\dd x\): take \(u=x\) (\(\dd u=\dd x\)), \(\dd v=e^x\dd x\) (\(v=e^x\)). Then \(\int x e^x\dd x=xe^x-\int e^x\dd x=xe^x-e^x+C=e^x(x-1)+C.\)

Other standard techniques (overview)

  • Trigonometric integrals & substitutions: use identities, or substitute \(x=\sin\theta,\tan\theta,\sec\theta\) to clear \(\sqrt{a^2\pm x^2}\).

  • Partial fractions: split a rational function \(\tfrac{P(x)}{Q(x)}\) into simple pieces, each integrable to a log or arctangent. (This is also how one inverts the logistic ODE to derive the sigmoid.)

  • Tables / computer algebra: for everything else, look it up or use .

Many perfectly innocent-looking integrals have no elementary antiderivative — most famously the Gaussian \(\int e^{-x^2}\dd x\), which cannot be written with elementary functions (its definite version over \(\R\) is the beautiful \(\sqrt{\pi}\)). This is not a failure of skill; such integrals are evaluated numerically or via special functions. In ML this is the norm, not the exception, which is why we lean on numerical integration and Monte Carlo.

Numerical integration

When no closed form exists, approximate the area directly. The trapezoidal rule replaces the curve by line segments; Simpson’s rule uses parabolas and is far more accurate for smooth integrands: \[\int_a^b f\,\dd x \approx \frac{\Delta x}{2}\big[f_0+2f_1+\cdots+2f_{n-1}+f_n\big]\ \text{(trapezoid)}.\] In high dimensions even these fail (the “curse of dimensionality”), and Monte Carlo integration — averaging the integrand at random sample points — becomes the only practical option. That is precisely how expectations are estimated in modern ML.

Improper integrals

An improper integral has an infinite limit or an unbounded integrand; it is defined as a limit of proper integrals, e.g. \[\int_a^{\infty} f(x)\,\dd x = \lim_{b\to\infty}\int_a^b f(x)\,\dd x,\] and converges if that limit is finite, otherwise diverges.

\(\displaystyle\int_1^\infty \frac{1}{x^2}\,\dd x=\lim_{b\to\infty}\Big[-\tfrac1x\Big]_1^b=\lim_{b\to\infty}\big(1-\tfrac1b\big)=1\) (converges), whereas \(\int_1^\infty\tfrac1x\,\dd x\) diverges. Whether the tail decays fast enough decides convergence — the same question that decides whether a probability distribution has a finite mean or variance.

Improper integrals are the natural habitat of probability. Densities live on all of \(\R\) (the Gaussian \(\tfrac{1}{\sqrt{2\pi}}e^{-x^2/2}\)), so normalization \(\int_{-\infty}^{\infty}p=1\), means \(\int x\,p\,\dd x\), and entropies \(-\int p\ln p\,\dd x\) are all improper integrals. Because most are not elementary (that Gaussian again), ML estimates them with Monte Carlo: replace \(\E[f(X)]=\int f\,p\,\dd x\) by the sample average \(\tfrac1N\sum_i f(x_i)\) with \(x_i\sim p\). Mini-batch training, dropout, and variational inference are all Monte Carlo estimates of integrals.

import numpy as np
from scipy import integrate
import sympy as sp

x = sp.symbols('x')
print(sp.integrate(x*sp.exp(x), x))        # x*e^x - e^x  (by parts)
print(sp.integrate(sp.exp(-x**2), (x, -sp.oo, sp.oo)))   # sqrt(pi)

# Monte Carlo estimate of E[X^2] for X ~ N(0,1)  (true value = 1)
samples = np.random.randn(1_000_000)
print(np.mean(samples**2))                 # ~1.0

Chapter summary

  • Substitution reverses the chain rule; by parts reverses the product rule (\(\int u\,\dd v=uv-\int v\,\dd u\)).

  • Trig substitution and partial fractions handle roots and rational functions; many integrals need tables/CAS.

  • Some integrals (e.g. \(\int e^{-x^2}\)) have no elementary antiderivative — use numerical or special-function methods.

  • Numerical integration (trapezoid/Simpson) and, in high dimensions, Monte Carlo approximate areas.

  • Improper integrals (infinite range/integrand) are limits; they govern normalization, means, and entropies in ML.