The limit is the idea that makes calculus rigorous. “Instantaneous” rate of change and “infinitely thin” rectangles both hide a limit: a quantity we approach but may never reach. This chapter builds limits intuitively and precisely, the laws for computing them, and the notion of continuity — an “unbroken” function — which guarantees the well-behaved landscapes that optimization relies on.

The limit, intuitively

We write \(\displaystyle\lim_{x\to a} f(x) = L\) to mean: as \(x\) gets arbitrarily close to \(a\) (from either side, but not equal to \(a\)), the value \(f(x)\) gets arbitrarily close to \(L\). The limit describes where the function is heading, regardless of what — if anything — happens exactly at \(a\).

The phrase “but not equal to \(a\)” is the whole point. A limit asks about the neighborhood of a point, not the point itself. This is exactly what lets us make sense of \(\frac{f(x+h)-f(x)}{h}\) as \(h\to0\): we never divide by zero; we watch where the ratio heads as \(h\) shrinks. The function may even be undefined at \(a\) and still have a perfectly good limit there.

Consider \(f(x)=\dfrac{x^2-1}{x-1}\). At \(x=1\) this is \(\tfrac{0}{0}\), undefined. But for \(x\neq1\), \(f(x)=\dfrac{(x-1)(x+1)}{x-1}=x+1\), so \(\displaystyle\lim_{x\to1}f(x)=2\). The graph is the line \(y=x+1\) with a single hole at \((1,2)\) — the limit fills in the hole.

One-sided and infinite limits

Sometimes the approach direction matters. The left limit \(\lim_{x\to a^-}f(x)\) and right limit \(\lim_{x\to a^+}f(x)\) consider \(x<a\) and \(x>a\) separately; the two-sided limit exists only when both agree. Limits can also be infinite (a vertical asymptote, \(f\to\pm\infty\)) or taken at infinity (\(x\to\pm\infty\), describing end behavior). Limits at infinity are how we describe the tails of distributions and the saturation of activation functions.

Computing limits: the limit laws

For most functions, limits are computed by the obvious algebra, provided no division by zero occurs.

If \(\lim_{x\to a}f\) and \(\lim_{x\to a}g\) exist, then limits distribute over sums, products, and quotients (denominator limit nonzero): \[\lim(f\pm g)=\lim f\pm\lim g,\quad \lim(fg)=\lim f\cdot\lim g,\quad \lim\frac{f}{g}=\frac{\lim f}{\lim g}.\] For a continuous function, \(\lim_{x\to a}f(x)=f(a)\) — you may simply “plug in.”

When plugging in gives an indeterminate form like \(\tfrac00\) or \(\tfrac{\infty}{\infty}\), the limit is not automatic: it must be resolved by algebra (as above), by the squeeze theorem, or later by L’Hôpital’s rule (Chapter [ch:appderiv]).

If \(g(x)\le f(x)\le h(x)\) near \(a\) and \(\lim_{x\to a}g=\lim_{x\to a}h=L\), then \(\lim_{x\to a}f=L\). Trapping a function between two others that meet forces it to the same limit; this is how one proves the foundational \(\displaystyle\lim_{x\to0}\frac{\sin x}{x}=1\).

The precise definition (epsilon–delta)

The intuitive “arbitrarily close” is made exact by the \(\varepsilon\)–\(\delta\) definition — the logical bedrock of analysis.

\(\displaystyle\lim_{x\to a}f(x)=L\) means: for every tolerance \(\varepsilon>0\) there is a \(\delta>0\) such that whenever \(0<|x-a|<\delta\), we have \(|f(x)-L|<\varepsilon\).

Read it as a challenge–response game. A skeptic demands that \(f(x)\) land within \(\varepsilon\) of \(L\); you must produce a radius \(\delta\) around \(a\) guaranteeing it. If you can answer every \(\varepsilon\), the limit truly is \(L\). You will rarely compute with \(\varepsilon\)–\(\delta\) directly, but it is what makes every later theorem trustworthy.

Continuity

A function \(f\) is continuous at \(a\) if \(\displaystyle\lim_{x\to a}f(x)=f(a)\) — the limit exists, the function is defined there, and the two agree. A function continuous at every point of an interval is continuous on it: its graph has no holes, jumps, or breaks — you can draw it without lifting your pen.

Sums, products, quotients (nonzero denominator), and compositions of continuous functions are continuous, so polynomials, exponentials, logs (on their domain), and sines/cosines are all continuous — which is why “plug in” usually works.

If \(f\) is continuous on \([a,b]\) and \(N\) lies between \(f(a)\) and \(f(b)\), then \(f(c)=N\) for some \(c\in[a,b]\). A continuous function cannot skip values: to get from one height to another it must pass through every height in between.

The IVT guarantees that root-finding methods like bisection work, and the related Extreme Value Theorem (a continuous function on a closed interval attains a maximum and minimum) is what makes optimization well-posed.

A limit existing does not require the function to be defined at the point, and a function being defined does not guarantee continuity. Watch for removable holes (\(\tfrac00\) that cancels), jumps (one-sided limits disagree), and blow-ups (vertical asymptotes). In ML, a subtler issue is functions that are continuous but not differentiable at a point — ReLU \(\max(0,x)\) has a corner at \(0\) — which is fine for forward passes but needs care (subgradients) when differentiating.

Continuity is the unspoken assumption that makes learning possible. Gradient-based training implicitly assumes the loss surface is continuous (and almost-everywhere differentiable) so that a small parameter change produces a small, predictable change in loss — letting us follow slopes downhill. Limits at infinity describe saturation: the sigmoid flattens to \(0\) and \(1\), which both enables probabilistic outputs and causes vanishing gradients when units saturate. The limit defining the derivative, next chapter, is the literal foundation of backpropagation.

import numpy as np
# sin(x)/x -> 1 as x -> 0 (a squeeze-theorem classic)
for x in [1, 0.1, 0.01, 1e-6]:
    print(x, np.sin(x)/x)
# bisection relies on the Intermediate Value Theorem
f = lambda x: x**2 - 2          # root at sqrt(2)
lo, hi = 1.0, 2.0
for _ in range(40):
    mid = (lo+hi)/2
    lo, hi = (mid, hi) if f(mid) < 0 else (lo, mid)
print((lo+hi)/2)                # ~ 1.41421356

Chapter summary

  • A limit \(\lim_{x\to a}f(x)=L\) describes where \(f\) heads near \(a\), ignoring the value at \(a\) itself.

  • Two-sided limits need agreeing one-sided limits; limits can be infinite or taken at infinity (saturation, tails).

  • Use the limit laws; resolve indeterminate forms by algebra, the squeeze theorem, or L’Hôpital later.

  • The \(\varepsilon\)–\(\delta\) definition makes “arbitrarily close” precise and underpins every theorem.

  • Continuity (\(\lim_{x\to a}f=f(a)\)) gives unbroken graphs; the IVT and EVT make root-finding and optimization well-posed.