Calculus studies how things change and how things accumulate. Before we can speak of change, we need the object that changes: the function. This chapter sets up functions and the handful of “essential” ones that appear everywhere, then introduces the two questions — the slope of a curve and the area under it — that the entire subject is built to answer.

What is a function?

A function \(f\) is a rule that assigns to each input \(x\) in a set called the domain exactly one output \(f(x)\). The set of outputs is the range. We write \(f:\R\to\R\) for a function taking a real number to a real number.

Think of a function as a machine: feed in \(x\), get out \(f(x)\) — and crucially, the same input always yields the same output (that is the “exactly one” clause). A function can be described four ways: by a formula (\(f(x)=x^2\)), a graph, a table of values, or a verbal rule. Fluency means moving between these views; calculus mostly reasons with the graph and the formula.

A catalog of essential functions

A small library of functions covers almost everything in machine learning:

  • Linear \(f(x)=mx+b\): constant slope \(m\); the simplest model and the local picture of every differentiable function.

  • Power \(f(x)=x^n\) and polynomials: sums of powers.

  • Exponential \(f(x)=e^x\): the function equal to its own derivative; models growth, decay, and (through \(e^{-x}\)) the tails of probability distributions.

  • Logarithm \(f(x)=\ln x\): the inverse of \(e^x\); turns products into sums and powers into products, which is why log-likelihoods are everywhere in ML.

  • Trigonometric \(\sin x,\cos x\): periodic; the basis of Fourier analysis and positional encodings.

Building new functions from old

Functions combine by addition, multiplication, and especially composition: \((f\circ g)(x)=f(g(x))\), “do \(g\), then \(f\).” Composition is the structural backbone of deep learning — a neural network is a deep composition of simple functions — and the chain rule (Chapter [ch:rules]) exists precisely to differentiate compositions.

A function is one-to-one if different inputs give different outputs; only then does it have an inverse \(f^{-1}\) that undoes it: \(f^{-1}(f(x))=x\). Graphically, \(f^{-1}\) is the reflection of \(f\) across the line \(y=x\) (as with \(e^x\) and \(\ln x\) above).

The two big problems

Calculus was invented to solve two geometric problems that turned out to be inverse to each other.

The tangent problem. Given the graph of \(f\), what is its slope at a point — the instantaneous rate of change? A straight line has an obvious slope; a curve does not, because the slope changes everywhere. Resolving this gives the derivative (differential calculus).

The area problem. Given the graph of \(f\), what is the area between it and the \(x\)-axis over an interval — the total accumulation? Rectangles approximate it; making them infinitely thin gives the exact answer. Resolving this gives the integral (integral calculus).

The two problems are secretly the same problem viewed from opposite ends. If \(f(t)\) is your position at time \(t\), the slope of the position graph is your velocity (a derivative), and the area under the velocity graph is the distance you travelled (an integral). Differentiation and integration undo each other — a fact so central it is called the Fundamental Theorem of Calculus (Chapter [ch:integral]).

Rate of change: the average that becomes instantaneous

The slope of \(f\) between two points is the average rate of change: \[\frac{\Delta y}{\Delta x}=\frac{f(x+h)-f(x)}{h}.\] This is the slope of the secant line through the two points. The whole trick of differential calculus is to let \(h\to0\): the secant lines approach the tangent line, and the average rate becomes the instantaneous rate. Making that limit precise is the job of Chapter [ch:limits].

Functions are the models of machine learning, and composition is their architecture. A neural network is a function \(f_{\vtheta}\) with millions of parameters \(\vtheta\), built by composing linear maps with simple nonlinearities (activation functions) such as the sigmoid \(\sigma(x)=1/(1+e^{-x})\), \(\tanh\), and ReLU \(\max(0,x)\) — all from the catalog above. Training adjusts \(\vtheta\) so the function fits data, and that adjustment is driven by rates of change (derivatives) of a loss with respect to each parameter. The exponential and logarithm in particular run the softmax and cross-entropy that almost every classifier ends with.

import numpy as np
f = lambda x: 0.5*x**2
# average rate of change over [1.5, 1.5+h] approaches the slope 1.5 as h->0
for h in [1, 0.1, 0.01, 1e-4]:
    print(h, (f(1.5+h) - f(1.5)) / h)     # -> 1.5
sigmoid = lambda x: 1/(1+np.exp(-x))       # a key ML activation function
print(sigmoid(np.array([-2.0, 0.0, 2.0])))

Chapter summary

  • A function assigns one output to each input; describe it by formula, graph, table, or words.

  • Know the essential functions: linear, power/polynomial, \(e^x\), \(\ln x\), \(\sin/\cos\) — and composition \(f(g(x))\).

  • Calculus answers two inverse questions: the tangent (slope/derivative) and the area (integral).

  • The derivative is the limit of average rates of change \(\frac{f(x+h)-f(x)}{h}\) as \(h\to0\).

  • ML models are functions built by composition; activations come straight from this catalog.