The derivative is the central object of differential calculus and the single most important concept for machine learning. It answers the tangent problem: the instantaneous rate at which a function changes. Once we have it, “which way is downhill?” — the question every learning algorithm asks — has a precise, computable answer.

Definition: the limit of a difference quotient

The derivative of \(f\) at \(x\) is \[f'(x) = \lim_{h\to 0}\frac{f(x+h)-f(x)}{h},\] provided the limit exists. The quotient inside is the slope of the secant line through \((x,f(x))\) and \((x+h,f(x+h))\); the limit is the slope of the tangent line at \(x\).

As \(h\) shrinks, the secant line pivots toward the tangent line, and its slope approaches the derivative. The derivative is the instantaneous rate of change — the speedometer reading rather than the trip’s average speed — and equivalently the slope of the curve at that point. It is also the best linear approximation: near \(x\), \(f(x+h)\approx f(x)+f'(x)\,h\).

For \(f(x)=x^2\): \[f'(x)=\lim_{h\to0}\frac{(x+h)^2-x^2}{h}=\lim_{h\to0}\frac{2xh+h^2}{h}=\lim_{h\to0}(2x+h)=2x.\] The slope of \(y=x^2\) at \(x\) is \(2x\): zero at the bottom (\(x=0\)), increasingly steep as \(|x|\) grows.

Notation

The same idea wears several notations, each handy in context: \[f'(x)\quad=\quad \dydx{y}{x}\quad=\quad \ddx{x}f(x)\quad=\quad Df(x)\quad=\quad \dot{y}\ (\text{for time}).\] The Leibniz form \(\dydx{y}{x}\) is suggestive: a ratio of an infinitesimal output change \(\dd y\) to an input change \(\dd x\). It is not literally a fraction, but it behaves like one in ways that make the chain rule and substitution feel natural.

Differentiability and continuity

If \(f\) is differentiable at \(x\), then \(f\) is continuous at \(x\). The converse is false: continuity does not imply differentiability.

Differentiable means locally straight: zoom in far enough and the graph is indistinguishable from its tangent line. A function can be continuous (unbroken) yet fail this at sharp corners or vertical tangents. The absolute value \(|x|\) is continuous everywhere but has no derivative at \(0\) — the left slope is \(-1\), the right slope is \(+1\), and they disagree.

The ML-critical example is ReLU, \(\max(0,x)\): continuous, differentiable everywhere except the corner at \(0\). Frameworks simply pick a subgradient (usually \(0\)) at the kink, which works fine in practice. The lesson: differentiability can fail at isolated points without breaking gradient-based learning, but you should know where the kinks are.

Higher-order derivatives

Differentiating the derivative gives the second derivative \(f''(x)=\dfrac{\dd^2 y}{\dd x^2}\), the rate of change of the slope — i.e. curvature/concavity. If \(f(t)\) is position, \(f'\) is velocity and \(f''\) is acceleration. The sign of \(f''\) tells you whether the graph bends up (convex, \(f''>0\)) or down (concave, \(f''<0\)), which is exactly the second-derivative test for minima and maxima (Chapter [ch:appderiv]) and, in many variables, the curvature of the loss surface (the Hessian).

The derivative as a function and as a rate

Computing \(f'(x)\) at every \(x\) produces a new function, the derivative function. Its readings are rates of change with units of “output per input”: a velocity (m/s), a marginal cost ($/unit), a reaction rate, or — in ML — the sensitivity of a loss to a parameter. That last reading is the one that matters most to us.

Training a model means minimizing a loss \(L(\vtheta)\). The derivative \(\partial L/\partial\theta_i\) answers: “if I increase parameter \(\theta_i\) a little, does the loss go up or down, and how fast?” Collect these into the gradient (Chapter [ch:multivariable]) and you know the steepest-descent direction; step against it and the model improves. This is gradient descent, and it is nothing but the derivative of Chapter 1 applied to millions of parameters at once. The limit definition is also how numerical gradient checks are computed to verify backpropagation.

import numpy as np
f  = lambda x: x**2
df = lambda x: 2*x                  # exact derivative

# central finite difference approximates the derivative
def numeric_deriv(f, x, h=1e-5):
    return (f(x+h) - f(x-h)) / (2*h)

print(numeric_deriv(f, 3.0), df(3.0))   # ~6.0, 6.0
# corner of |x|: left and right slopes disagree at 0 (not differentiable)
g = np.abs
print(numeric_deriv(g, 0.0))            # ~0 by symmetry, but slope undefined at 0

Chapter summary

  • The derivative \(f'(x)=\lim_{h\to0}\frac{f(x+h)-f(x)}{h}\) is the tangent slope / instantaneous rate of change.

  • It is the best local linear approximation: \(f(x+h)\approx f(x)+f'(x)h\).

  • Notations: \(f'\), \(\dydx{y}{x}\), \(Df\); differentiable \(\Rightarrow\) continuous, but not conversely (corners like \(|x|\), ReLU).

  • The second derivative measures curvature/concavity (convex vs concave); it drives the minimum/maximum tests.

  • In ML the derivative is the sensitivity of the loss to a parameter — the basis of gradient descent.