Now that we can compute derivatives, we put them to work. Derivatives find the highest and lowest points of a function (optimization), approximate complicated functions by simple ones (linearization and Newton’s method), and resolve stubborn limits (L’Hôpital). Optimization in particular is the bridge to machine learning: training a model is minimizing a function.
Extrema and critical points
A function has a local maximum (or minimum) at \(c\) if \(f(c)\) is the largest (smallest) value nearby. A critical point is where \(f'(c)=0\) or \(f'\) does not exist. By Fermat’s theorem, every interior local extremum is a critical point.
At a smooth peak or valley the tangent line is horizontal: \(f'(c)=0\). So the search for maxima and minima begins by solving \(f'(x)=0\). This is the one-variable seed of all optimization — and in many variables it becomes “set the gradient to zero,” the condition every trained model approximately satisfies.
Classifying critical points
First-derivative test: if \(f'\) changes \(+\to-\) at \(c\), it is a local max; \(-\to+\), a local min.
Second-derivative test: if \(f'(c)=0\) and \(f''(c)>0\) (concave up, a bowl) it is a local min; \(f''(c)<0\) (concave down) a local max; \(f''(c)=0\) is inconclusive.
The second-derivative test is the one-variable ancestor of the Hessian test in many variables (Chapter [ch:multivariable]), which distinguishes minima, maxima, and saddle points of a loss surface.
The Mean Value Theorem
If \(f\) is continuous on \([a,b]\) and differentiable on \((a,b)\), then there is a point \(c\) where the instantaneous rate equals the average rate: \[f'(c) = \frac{f(b)-f(a)}{b-a}.\]
Over any trip, at some instant your speedometer reads exactly your average speed. The MVT is the theoretical workhorse behind many results: it implies that \(f'>0\) on an interval forces \(f\) to increase there, that a zero derivative everywhere means a constant function, and it underlies the error bounds for the approximation methods below.
Monotonicity, concavity, and curve sketching
The first derivative’s sign tells you where \(f\) rises (\(f'>0\)) or falls (\(f'<0\)); the second derivative’s sign tells you where it is convex (\(f''>0\)) or concave (\(f''<0\)), with inflection points where concavity flips. Together they let you sketch a curve’s qualitative shape — and, more usefully for us, they describe the geometry of loss functions: convex regions have a single basin, while non-convex regions have multiple basins and saddles.
Optimization in practice
Real optimization problems ask for the largest or smallest value of some quantity subject to constraints. The recipe: write the objective as a function of one variable, differentiate, solve \(f'=0\), and check endpoints and the second derivative.
Of all rectangles with perimeter \(20\), which has the largest area? With sides \(x\) and \(10-x\), area \(A(x)=x(10-x)=10x-x^2\). Then \(A'(x)=10-2x=0\Rightarrow x=5\), and \(A''=-2<0\) confirms a maximum. The square (\(5\times5\)) wins. Optimization problems in ML are vastly higher-dimensional, but the logic — differentiate, find where the derivative vanishes — is identical.
Linear approximation and differentials
Because a differentiable function is locally straight, the tangent line is an excellent local stand-in: \[f(x)\approx f(a) + f'(a)(x-a)\qquad\text{(linearization at }a\text{)}.\] This is the order-1 Taylor approximation (Chapter [ch:series]). It is the basis of error propagation, of the “local linear model” view of a neuron, and — iterated — of Newton’s method.
Newton’s method
To solve \(f(x)=0\), follow the tangent line to where it crosses the axis and repeat: \[x_{n+1} = x_n - \frac{f(x_n)}{f'(x_n)}.\]
Newton’s method replaces a hard equation by its tangent-line approximation at each step and solves that instead. When it works it is breathtakingly fast (the number of correct digits roughly doubles each iteration). Applied to \(f'=0\) instead of \(f=0\), it becomes Newton’s method for optimization, \(x_{n+1}=x_n-f'(x_n)/f''(x_n)\) — which in many variables uses the Hessian and is the ancestor of second-order optimizers (Chapter [ch:calcmlai]).
L’Hôpital’s rule
For indeterminate forms \(\tfrac00\) or \(\tfrac{\infty}{\infty}\), derivatives rescue the limit: \[\lim_{x\to a}\frac{f(x)}{g(x)} = \lim_{x\to a}\frac{f'(x)}{g'(x)}\] when the right-hand limit exists. For instance \(\lim_{x\to0}\frac{\sin x}{x}=\lim_{x\to0}\frac{\cos x}{1}=1\). It quantifies which of two competing quantities wins a race to \(0\) or \(\infty\) — useful for analyzing asymptotic rates, including how fast algorithms converge.
This chapter is the conceptual core of model training. Learning is optimization: minimize a loss \(L(\vtheta)\). The condition \(\nabla L=\vzero\) (the many-variable “\(f'(c)=0\)”) characterizes the stationary points training seeks; the second-derivative/Hessian test classifies them as minima, maxima, or the saddle points that pervade deep networks. Linearization is the first-order model behind gradient descent’s step; Newton’s method is the second-order ideal that methods like L-BFGS and K-FAC approximate. Even the learning-rate intuition — small steps along the tangent — is the linear approximation of this chapter in disguise.
import numpy as np
# Newton's method to solve f(x)=0 for f(x)=x^2-2 (root sqrt(2))
f, df = lambda x: x**2-2, lambda x: 2*x
x = 1.0
for _ in range(6):
x = x - f(x)/df(x)
print(x) # 1.4142135623730951
# 1-D gradient descent to MINIMIZE f(x)=(x-3)^2 (uses f', not f)
g = lambda x: 2*(x-3)
x, lr = 0.0, 0.1
for _ in range(60):
x -= lr*g(x)
print(x) # -> 3.0Chapter summary
Interior extrema occur at critical points (\(f'=0\)); classify with the first- or second-derivative test.
The MVT links average and instantaneous rates and powers many proofs and error bounds.
Sign of \(f'\) gives increase/decrease; sign of \(f''\) gives convex/concave and inflection points.
Linearization \(f(x)\approx f(a)+f'(a)(x-a)\) and Newton’s method approximate functions and find roots/optima fast.
L’Hôpital resolves \(\tfrac00\), \(\tfrac\infty\infty\); all of this is the one-variable seed of ML optimization.