Machine-learning models depend on millions of parameters at once, so single-variable calculus is not enough. This chapter extends differentiation to functions of several variables. The star object is the gradient — the vector of partial derivatives that points uphill — because following its negative is exactly gradient descent. We also meet the multivariable chain rule, the Jacobian, and the Hessian, the three pillars of training and curvature analysis.
Functions of several variables
A function \(f:\R^n\to\R\) takes a vector \(\vx=(x_1,\dots,x_n)\) and returns a number — for example a loss \(L(\vtheta)\) of all the model’s parameters. Its graph is a surface (for \(n=2\)) or hypersurface; we visualize it with contour lines (level sets where \(f\) is constant), exactly like elevation lines on a topographic map.
Partial derivatives
The partial derivative \(\pd{f}{x_i}\) is the ordinary derivative of \(f\) with respect to \(x_i\), treating all other variables as constants: \[\pd{f}{x_i} = \lim_{h\to0}\frac{f(\dots,x_i+h,\dots)-f(\dots,x_i,\dots)}{h}.\] It measures the rate of change of \(f\) as you move along the \(x_i\)-axis alone.
For \(f(x,y)=x^2y+\sin y\): \(\ \pd{f}{x}=2xy\) (treat \(y\) as constant), and \(\pd{f}{y}=x^2+\cos y\) (treat \(x\) as constant).
The gradient
The gradient of \(f:\R^n\to\R\) collects all partial derivatives into a vector: \[\grad f = \left[\pd{f}{x_1},\,\pd{f}{x_2},\,\dots,\,\pd{f}{x_n}\right]^\top.\]
The gradient points in the direction of steepest ascent, and its magnitude is the rate of climb in that direction. Therefore \(-\grad f\) points in the direction of steepest descent. That one fact is the engine of nearly all machine learning: to minimize a loss, repeatedly step a little in the direction \(-\grad L\). The gradient is perpendicular to the contour lines — always pointing “straight uphill” across them.
Directional derivatives
The rate of change of \(f\) in an arbitrary unit direction \(\vu\) is the directional derivative \[D_{\vu}f = \grad f\cdot\vu = \norm{\grad f}\cos\theta.\] It is largest when \(\vu\) aligns with \(\grad f\) (\(\theta=0\)) — confirming that the gradient is the steepest-ascent direction — and zero along the contour (\(\theta=90^\circ\)). The dot-product form ties this directly to the linear-algebra notion of alignment.
The tangent plane and linearization
Just as a curve has a tangent line, a surface has a tangent plane, giving the multivariable linear approximation: \[f(\vx+\Delta)\approx f(\vx) + \grad f(\vx)\cdot\Delta.\] This first-order model is the precise justification of a gradient-descent step: locally, the loss looks linear, and the steepest decrease is along \(-\grad f\).
The multivariable chain rule and the Jacobian
When \(f\) depends on variables that themselves depend on others, rates combine through the multivariable chain rule. For a vector-valued map \(\vf:\R^n\to\R^m\), all first-order information is packed into the Jacobian matrix: \[\mathbf{J} = \pd{\vf}{\vx},\qquad J_{ij}=\pd{f_i}{x_j}\quad(m\times n).\]
The Jacobian is the best linear approximation of a vector-valued function: locally \(\vf(\vx+\Delta)\approx\vf(\vx)+\mathbf{J}\Delta\). For a chain of maps the Jacobians multiply, exactly as one-variable derivatives did. Backpropagation is this chain rule applied to a deep composition — a product of layer Jacobians, evaluated efficiently from the output backward (the gradient is one row, so we propagate vectors, never the huge full matrices).
Second derivatives: the Hessian
The Hessian of \(f:\R^n\to\R\) is the symmetric matrix of second partial derivatives, \[\mathbf{H}_{ij} = \frac{\partial^2 f}{\partial x_i\,\partial x_j}.\] It encodes the curvature of \(f\) in every direction at once (assuming the mixed partials are continuous, \(\mathbf{H}\) is symmetric — Clairaut’s theorem).
The Hessian is the multivariable second-derivative test. At a point where \(\grad f=\vzero\): if \(\mathbf{H}\) is positive definite (all eigenvalues \(>0\)) it is a local minimum; negative definite, a maximum; indefinite (mixed-sign eigenvalues), a saddle point. The eigenvalues of the Hessian are the curvatures along the principal directions, and their ratio — the condition number — governs how fast (or how painfully slowly) gradient descent converges.
The second-order Taylor expansion ties it together: \[f(\vx+\Delta)\approx f(\vx)+\grad f\cdot\Delta+\tfrac12\,\Delta^\top \mathbf{H}\,\Delta.\] Newton’s method minimizes this quadratic model exactly, stepping \(\Delta=-\mathbf{H}^{-1}\grad f\).
This chapter is the mathematical core of training. The gradient \(\grad_{\vtheta}L\) gives the descent direction; gradient descent is \(\vtheta\leftarrow\vtheta-\eta\,\grad_{\vtheta}L\). The Jacobian and the multivariable chain rule are backpropagation. The Hessian explains why deep loss surfaces are hard: in high dimensions, critical points are overwhelmingly saddles, not local minima, and ill-conditioned Hessians cause the zig-zagging that momentum and Adam are designed to fix. Newton/quasi-Newton methods (L-BFGS, K-FAC) approximate \(\mathbf{H}^{-1}\) because the true Hessian is far too large to form for billions of parameters. Chapter [ch:calcmlai] makes all of this explicit.
import torch
# gradient and Hessian of f(x,y) = x^2 + 1.6 y^2 at (1.6, 1.4)
def f(v): return v[0]**2 + 1.6*v[1]**2
v = torch.tensor([1.6, 1.4], requires_grad=True)
g = torch.autograd.functional.jacobian(f, v) # gradient = [2x, 3.2y]
H = torch.autograd.functional.hessian(f, v) # [[2,0],[0,3.2]]
print(g) # tensor([3.2000, 4.4800])
print(H) # diag(2, 3.2): eigenvalues = curvaturesChapter summary
Partial derivatives differentiate in one variable at a time; the gradient \(\grad f\) stacks them.
\(\grad f\) points in the direction of steepest ascent and is \(\perp\) to contours; \(-\grad f\) is steepest descent.
The directional derivative is \(\grad f\cdot\vu\); the tangent plane gives the first-order approximation.
The Jacobian (first derivatives of a vector map) multiplies along chains — this is backpropagation.
The Hessian (second derivatives) gives curvature; its definiteness classifies minima/maxima/saddles and sets convergence speed.