Computing every derivative from the limit definition would be exhausting. Fortunately, a small set of rules lets us differentiate almost any function mechanically. The most important of these — the chain rule — is the mathematical heart of backpropagation, so this chapter is worth real fluency.
The building-block derivatives
Memorize these; everything else is built from them.
| \(f(x)\) | \(f'(x)\) | \(f(x)\) | \(f'(x)\) |
|---|---|---|---|
| \(c\) (constant) | \(0\) | \(\sin x\) | \(\cos x\) |
| \(x^n\) | \(n x^{n-1}\) | \(\cos x\) | \(-\sin x\) |
| \(e^x\) | \(e^x\) | \(\tan x\) | \(\sec^2 x\) |
| \(a^x\) | \(a^x\ln a\) | \(\ln x\) | \(1/x\) |
| \(\sqrt{x}\) | \(\tfrac{1}{2\sqrt x}\) | \(\log_a x\) | \(1/(x\ln a)\) |
Two facts deserve a star. First, \(\ddx{x}e^x=e^x\): the exponential is its own derivative, the only function (up to scaling) with this property — which is why it governs growth, decay, and the math of probability. Second, \(\ddx{x}\ln x=\tfrac1x\): the logarithm converts the multiplicative into the additive, the reason log-likelihoods turn products of probabilities into sums we can differentiate term by term.
The algebraic rules
For differentiable \(f,g\) and constant \(c\): \[(cf)' = c f',\qquad (f\pm g)' = f' \pm g',\] \[\textbf{Product: }(fg)' = f'g + fg',\qquad \textbf{Quotient: }\left(\frac{f}{g}\right)' = \frac{f'g - fg'}{g^2}.\]
The derivative is linear, so sums split cleanly — but products and quotients do not. A frequent error is writing \((fg)'=f'g'\); it is not. Use the product rule “derivative of first times second, plus first times derivative of second.” Sign errors in the quotient rule (numerator order matters) are the other classic slip.
\(\ddx{x}\big(x^2 e^x\big) = (2x)e^x + x^2 e^x = e^x(x^2+2x)\), by the product rule.
The chain rule
The chain rule differentiates compositions — and since deep networks are deep compositions, it is the rule that makes deep learning possible.
If \(y=f(u)\) and \(u=g(x)\), then \[\dydx{y}{x} = \dydx{y}{u}\cdot\dydx{u}{x} = f'(g(x))\,g'(x).\] “Differentiate the outer function (leaving the inside alone), then multiply by the derivative of the inside.”
The Leibniz notation makes it look like the \(\dd u\)’s cancel, and that is exactly the right mental model: rates of change multiply along a chain. If \(y\) changes \(3\times\) as fast as \(u\), and \(u\) changes \(2\times\) as fast as \(x\), then \(y\) changes \(6\times\) as fast as \(x\). Stack a hundred such links and you get a product of a hundred derivatives — the structure of backpropagation, and the origin of vanishing/exploding gradients when those factors are persistently \(<1\) or \(>1\).
\(\ddx{x}\, e^{-x^2}\): outer \(e^{u}\) with \(u=-x^2\), so derivative \(=e^{-x^2}\cdot(-2x)=-2x\,e^{-x^2}\).
For \(\sigma(x)=\dfrac{1}{1+e^{-x}}\), the chain and quotient rules give the famously clean \[\sigma'(x)=\sigma(x)\big(1-\sigma(x)\big).\] The derivative is expressed entirely through the output — so a network can reuse the forward-pass value during the backward pass, saving computation. This identity is a small reason sigmoids were historically popular.
Implicit and logarithmic differentiation
When \(y\) is tangled with \(x\) in an equation (e.g. \(x^2+y^2=1\)), implicit differentiation differentiates both sides with respect to \(x\), treating \(y\) as a function of \(x\) and applying the chain rule to \(y\)-terms, then solves for \(\dydx{y}{x}\). Logarithmic differentiation — take \(\ln\) of both sides first — tames products, quotients, and variable exponents like \(x^x\), converting them to sums before differentiating. Both are routine once the chain rule is solid.
Derivatives of inverse functions
If \(f\) and \(f^{-1}\) are inverses, their slopes are reciprocals at corresponding points: \[\big(f^{-1}\big)'(y)=\frac{1}{f'(x)}\quad\text{where }y=f(x).\] This yields \(\ddx{x}\ln x=\tfrac1x\) from \(\ddx{x}e^x=e^x\), and the derivatives of inverse trig functions — and it is the calculus behind the change-of-variables and normalizing-flow densities mentioned later.
The chain rule is backpropagation. A neural network composes layers \(f_L\circ\cdots\circ f_1\); the gradient of the scalar loss with respect to early parameters is a product of per-layer derivatives, computed efficiently from output back to input (Chapter [ch:calcmlai]). Activation derivatives from this chapter are used at every step: \(\sigma'=\sigma(1-\sigma)\), \(\tanh'=1-\tanh^2\), and \(\text{ReLU}'(x)=\mathbb{1}[x>0]\). Persistent factors below \(1\) cause vanishing gradients; above \(1\), exploding gradients — both are direct consequences of the chain rule’s multiplication.
import torch
x = torch.tensor(0.7, requires_grad=True)
y = torch.sigmoid(x)
y.backward() # autodiff applies the chain rule
print(x.grad, (y*(1-y)).detach()) # match: sigma'(x) = sigma(1-sigma)
x2 = torch.tensor(1.3, requires_grad=True)
(torch.exp(-x2**2)).backward()
print(x2.grad, (-2*1.3*torch.exp(-torch.tensor(1.3)**2))) # -2x e^{-x^2}Chapter summary
Memorize the building blocks; note \(\ddx{x}e^x=e^x\) and \(\ddx{x}\ln x=\tfrac1x\).
Derivatives are linear; use the product \((fg)'=f'g+fg'\) and quotient rules (not \(f'g'\)!).
The chain rule \(\dydx{y}{x}=f'(g(x))g'(x)\) multiplies rates along a composition — the basis of backprop.
Implicit, logarithmic, and inverse-function differentiation extend the toolkit.
Activation derivatives (\(\sigma'=\sigma(1-\sigma)\), \(\tanh'=1-\tanh^2\), ReLU\('=\mathbb{1}[x>0]\)) and vanishing/exploding gradients come straight from these rules.