This chapter gathers the calculus that most directly becomes machine learning: how to find minima of functions of many variables (with and without constraints), the gradient-descent dynamics that training uses, and the differential equations that describe how systems — and increasingly, generative models — evolve in continuous time. It is the launch pad for the capstone.
Unconstrained optimization
To minimize \(f:\R^n\to\R\), the multivariable analogue of \(f'=0\) is \[\grad f(\vx^\star)=\vzero \quad\text{(first-order/stationarity condition)},\] and the Hessian classifies the stationary point: positive definite \(\Rightarrow\) minimum, negative definite \(\Rightarrow\) maximum, indefinite \(\Rightarrow\) saddle (Chapter [ch:multivariable]). Solving \(\grad f=\vzero\) in closed form is usually impossible for the functions ML cares about, so we descend iteratively instead.
Gradient descent
Gradient descent minimizes \(f\) by repeatedly stepping against the gradient: \[\vx_{t+1} = \vx_t - \eta\,\grad f(\vx_t),\] where \(\eta>0\) is the learning rate (step size).
Picture a ball rolling downhill on the surface \(f\). The gradient points uphill, so \(-\grad f\) points to the locally fastest decrease; a small step that way lowers \(f\). The learning rate sets the step length: too small and progress crawls; too large and you overshoot and may diverge. The right scale is tied to curvature — specifically the largest eigenvalue of the Hessian — which is why ill-conditioned problems (long, narrow valleys) make plain gradient descent zig-zag.
Convexity
\(f\) is convex if its graph lies below every chord; equivalently (when smooth) its Hessian is positive semidefinite everywhere. For a convex function, every local minimum is global, and \(\grad f=\vzero\) pinpoints it.
Convexity is the dividing line between “easy” and “hard” optimization. Convex losses (linear/logistic regression, SVMs) have a single basin: gradient descent provably reaches the global minimum. Deep networks are non-convex — many basins, many saddles — so we settle for good local minima. Remarkably, in very high dimensions most stationary points are saddles rather than bad local minima, and momentum/adaptive methods escape them well enough to train enormous models.
Constrained optimization: Lagrange multipliers
To optimize \(f(\vx)\) subject to a constraint \(g(\vx)=0\), introduce a multiplier \(\lambda\) and solve \[\grad f = \lambda\,\grad g,\qquad g(\vx)=0.\]
At a constrained optimum, you cannot improve \(f\) without violating the constraint — which happens exactly when the contour of \(f\) is tangent to the constraint surface, i.e. their gradients are parallel. The multiplier \(\lambda\) measures how hard the constraint “pushes back” (the shadow price). Inequality constraints generalize this to the KKT conditions, the backbone of support vector machines and constrained ML.
Maximize entropy \(-\sum_i p_i\ln p_i\) subject to \(\sum_i p_i=1\): the Lagrange condition gives \(p_i=\) const, i.e. the uniform distribution — the maximum-entropy principle. Adding a mean constraint yields the exponential/softmax family.
Differential equations
A differential equation relates a function to its own derivatives — the language of change over time.
An ordinary differential equation (ODE) has the form \(\dydx{y}{t}=F(t,y)\). A separable equation \(\dydx{y}{t}=g(t)h(y)\) is solved by separating and integrating: \(\int\frac{\dd y}{h(y)}=\int g(t)\,\dd t\).
\(\dydx{y}{t}=ky\) gives \(y=y_0e^{kt}\) (unbounded growth/decay). Adding a saturation term, the logistic ODE \(\dydx{y}{t}=ky(1-y)\) integrates (via partial fractions) to the sigmoid \(y=\frac{1}{1+e^{-kt}}\) — the same S-curve used as a neural activation and as a probability.
When an ODE has no closed-form solution, Euler’s method steps along the tangent, \(y_{t+1}=y_t+\Delta t\,F(t,y_t)\) — structurally identical to a gradient-descent update, and the simplest of the numerical ODE solvers used inside neural ODEs and diffusion samplers.
A glimpse of the calculus of variations
Where ordinary optimization finds the best point, the calculus of variations finds the best function — the curve minimizing an integral functional, via the Euler–Lagrange equation. It underlies optimal control, the continuous-time view of residual networks, and the optimal-control perspective on diffusion-based generative models.
This chapter is essentially “the math of training,” one level before the capstone. Gradient descent and its variants (momentum, RMSProp, Adam) are exactly the iteration above with smarter step rules; convexity explains which models train reliably; Lagrange multipliers/KKT give SVMs and constrained objectives, and the max-entropy derivation yields softmax. ODEs appear as neural ODEs (a network defines \(\dydx{\vh}{t}\) and an ODE solver produces the output) and as the continuous-time formulation of diffusion models, whose samplers are ODE/SDE integrators. Euler’s method and gradient descent are the same tangent-following idea.
import numpy as np
# Gradient descent on a 2-D quadratic f(x) = x0^2 + 10 x1^2 (ill-conditioned)
grad = lambda x: np.array([2*x[0], 20*x[1]])
x, eta = np.array([5.0, 1.0]), 0.08
for _ in range(50):
x = x - eta*grad(x)
print(x) # -> near [0, 0]
# Euler's method for dy/dt = k y(1-y): recovers the logistic/sigmoid curve
y, k, dt = 0.01, 1.0, 0.05
ys = [y]
for _ in range(300):
y = y + dt*(k*y*(1-y)); ys.append(y)
print(round(ys[-1], 4)) # -> ~1.0 (saturates)Chapter summary
Minima satisfy \(\grad f=\vzero\) with positive-(semi)definite Hessian; solve iteratively by gradient descent \(\vx_{t+1}=\vx_t-\eta\grad f\).
Convex problems have one global basin; deep learning is non-convex (saddles dominate in high dimensions).
Lagrange multipliers (\(\grad f=\lambda\grad g\)) solve constrained problems; max-entropy yields the softmax family.
ODEs model continuous change; separable equations integrate directly, and Euler’s method steps along tangents.
These are the direct ancestors of Adam, SVMs, neural ODEs, and diffusion samplers.