Two ideas carry linear algebra the last mile into deep learning. First, tensors generalize vectors and matrices to higher-order arrays — the natural container for images, sequences, and batches. Second, matrix calculus extends derivatives to vector- and matrix-valued functions, giving the gradients and Jacobians that backpropagation multiplies together. This chapter is the bridge from “linear algebra” to “how neural networks actually train.”

Tensors: arrays of any order

A tensor (in the computational sense) is an \(n\)-dimensional array. Order 0 is a scalar, order 1 a vector, order 2 a matrix, and order \(\ge3\) a higher tensor. Its shape lists the size along each axis; e.g. a batch of RGB images is shape \((N, C, H, W)\).

Tensors are bookkeeping for “many matrices at once.” Deep learning almost never works with a single vector; it works with batches, channels, time steps, and attention heads stacked along extra axes. The operations are still fundamentally linear-algebraic — they are matrix multiplies applied along chosen axes — but the index bookkeeping needs a richer notation.

Broadcasting and einsum

Broadcasting stretches a smaller array across a larger one without copying (adding a bias vector to every row of a matrix). Einstein summation () names each axis and sums over repeated indices, expressing any contraction unambiguously: \[C_{ij}=\sum_k A_{ik}B_{kj}\ \Longleftrightarrow\ \texttt{einsum("ik,kj->ij", A, B)}.\] Batched matrix multiply, attention scores, and convolutions are all one away — the notation is the practical lingua franca of tensor code.

Tensor reshaping is where bugs hide. reinterprets the same data in row-major order; / reorders axes and changes the memory layout. Confusing them silently scrambles your data. Always reason about shapes (and print them); a shape mismatch caught early saves hours, and many “model not learning” bugs are really an axis that was summed or transposed wrongly.

Derivatives in many dimensions

Generalizing the scalar derivative, we organize partial derivatives into vectors and matrices.

For a scalar function \(f:\R^n\to\R\), the gradient is the vector of partial derivatives \(\nabla f=\big[\frac{\partial f}{\partial x_1},\dots,\frac{\partial f}{\partial x_n}\big]\tp\). For a vector function \(\vf:\R^n\to\R^m\), the Jacobian is the \(m\times n\) matrix \[\mathbf{J}_{ij}=\frac{\partial f_i}{\partial x_j}.\]

The gradient points in the direction of steepest increase of a scalar function — it is what gradient descent steps against. The Jacobian is the best linear approximation of a vector function near a point: locally, \(\vf(\vx+\Delta)\approx \vf(\vx)+\mathbf{J}\,\Delta\). For a linear map \(\vf(\vx)=\mA\vx\), the Jacobian is simply \(\mA\) itself — “the derivative of a linear map is the map.” Linear algebra and calculus meet exactly here.

Essential gradient identities

A few results cover most of machine learning (here \(\mA\) is constant, \(\vx\) the variable): \[\nabla_{\vx}(\va\tp\vx)=\va,\qquad \nabla_{\vx}(\vx\tp\mA\vx)=(\mA+\mA\tp)\vx,\qquad \nabla_{\vx}\norm{\vx}_2^2=2\vx.\] Applied to the least-squares loss \(L(\vx)=\norm{\mA\vx-\vb}_2^2\): \[\nabla_{\vx}L = 2\mA\tp(\mA\vx-\vb)=\vzero \ \Longrightarrow\ \mA\tp\mA\vx=\mA\tp\vb,\] recovering the normal equations of Chapter [ch:orthogonality] by pure calculus — the same answer the geometry gave.

The chain rule and backpropagation

A neural network is a composition \(\vf = \vf_L\circ\cdots\circ\vf_1\). The multivariable chain rule says the Jacobian of a composition is the product of the Jacobians: \[\frac{\partial L}{\partial \vx} = \frac{\partial L}{\partial \vf_L}\,\frac{\partial \vf_L}{\partial \vf_{L-1}}\cdots\frac{\partial \vf_1}{\partial \vx}.\]

Backpropagation is the chain rule evaluated right-to-left. Because the loss is scalar, the leftmost factor is a row vector, and multiplying by Jacobians from the left keeps a vector (never a giant matrix) flowing backward. Each step is a vector–Jacobian product (VJP): we never build the full Jacobian — for a layer with millions of inputs and outputs that matrix would be astronomically large. Instead we use a closed-form rule for “upstream gradient \(\mapsto\) downstream gradient.”

For \(\mathbf{Y}=\mX\mW\) with scalar loss \(L\) and upstream gradient \(\mathbf{G}=\partial L/\partial\mathbf{Y}\), the downstream gradients are obtained without forming any Jacobian: \[\frac{\partial L}{\partial \mW}=\mX\tp\mathbf{G},\qquad \frac{\partial L}{\partial \mX}=\mathbf{G}\,\mW\tp.\] Note the transposes and the order: the backward pass of a matrix multiply is two more matrix multiplies. This is the single most-executed gradient computation in deep learning.

Automatic differentiation can multiply the chain of Jacobians in two orders. Forward mode goes input-to-output and is efficient when there are few inputs. Reverse mode (backprop) goes output-to-input and computes the gradient with respect to all parameters in roughly one backward pass — ideal when there is one scalar loss and millions of parameters, exactly the deep-learning regime. That asymmetry is why every framework defaults to reverse-mode .

This chapter is the mechanism of training. Frameworks (PyTorch, JAX, TensorFlow) build a computation graph of tensor operations, attach a VJP rule to each, and call them in reverse to get \(\nabla_{\vtheta}L\) for every parameter; an optimizer then steps \(\vtheta\leftarrow\vtheta-\eta\nabla_{\vtheta}L\). Gradients are matrix products; activations are tensors; the whole forward and backward pass is linear algebra interleaved with cheap nonlinearities. Understanding Jacobians and VJPs lets you derive custom layers, debug exploding/vanishing gradients, and read the math of modern papers.

import torch
X = torch.randn(4, 3, requires_grad=False)
W = torch.randn(3, 2, requires_grad=True)
Y = X @ W
L = (Y**2).sum()
L.backward()                       # reverse-mode autodiff
print(torch.allclose(W.grad, X.T @ (2*Y)))   # matches X^T G, G = dL/dY
# einsum: batched matmul over a batch axis b
A = torch.randn(8, 4, 3); B = torch.randn(8, 3, 5)
C = torch.einsum("bij,bjk->bik", A, B)        # shape (8,4,5)

Chapter summary

  • Tensors are \(n\)-D arrays; broadcasting and express batched linear-algebra concisely; reason in shapes.

  • The gradient (steepest ascent) and Jacobian (best local linear map) organize derivatives; for \(\vf(\vx)=\mA\vx\), \(\mathbf{J}=\mA\).

  • Key identities give the least-squares gradient and re-derive the normal equations.

  • Backprop \(=\) chain rule right-to-left, a sequence of vector–Jacobian products; the Jacobian is never explicitly formed.

  • Reverse-mode autodiff yields all parameter gradients in one backward pass — the engine of deep learning.