A matrix is the workhorse object of linear algebra. This chapter develops the algebra of matrices — how to add, multiply, transpose, and invert them — always keeping sight of the three views from the preface: a matrix is a table, a transformation, and a collection of vectors.
Matrices as tables, transformations, and vector collections
A matrix \(\mA\in\R^{m\times n}\) is a rectangular array of numbers with \(m\) rows and \(n\) columns; the entry in row \(i\), column \(j\) is \(a_{ij}\). When \(m=n\) the matrix is square.
The three views, on one matrix \(\mA\):
Table: a spreadsheet — e.g. rows are customers, columns are features.
Transformation: a machine that takes a vector \(\vx\in\R^n\) and returns \(\mA\vx\in\R^m\), rotating/stretching/projecting space.
Vector collection: a bundle of \(n\) column vectors (or \(m\) row vectors). Its column space and row space (Chapter [ch:vectorspaces]) come from here.
Addition and scalar multiplication
These are componentwise and require matching shapes: \((\mA+\mB)_{ij}=a_{ij}+b_{ij}\) and \((c\mA)_{ij}=c\,a_{ij}\). They obey the obvious commutative/associative/distributive rules, so matrices of a fixed shape form a vector space themselves (Chapter [ch:vectorspaces]).
The matrix–vector product (three readings)
The product \(\mA\vx\) is the heart of the subject; read it three ways.
For \(\mA\in\R^{m\times n}\) and \(\vx\in\R^n\), the product \(\mA\vx\in\R^m\) can be computed as:
Row view: entry \(i\) is the dot product of row \(i\) of \(\mA\) with \(\vx\).
Column view: \(\mA\vx = x_1\va_1 + x_2\va_2 + \cdots + x_n\va_n\) — a linear combination of the columns \(\va_j\) of \(\mA\), weighted by the entries of \(\vx\).
Transformation view: \(\mA\) maps the input vector \(\vx\) to an output vector in \(\R^m\).
The column view, \(\mA\vx=\sum_j x_j\va_j\), is the most important identity in elementary linear algebra. It says the outputs of \(\mA\) are exactly the linear combinations of its columns — so the reachable outputs form the column space, and \(\mA\vx=\vb\) is solvable iff \(\vb\) lies there. Hold this beside the column picture of Chapter [ch:systems].
Matrix–matrix multiplication
For \(\mA\in\R^{m\times k}\) and \(\mB\in\R^{k\times n}\), the product \(\mC=\mA\mB\in\R^{m\times n}\) has entries \[c_{ij} = \sum_{\ell=1}^{k} a_{i\ell}\,b_{\ell j}\] — the dot product of row \(i\) of \(\mA\) with column \(j\) of \(\mB\). The inner dimensions must match (\(k=k\)); the outer dimensions give the result’s shape (\(m\times n\)).
Matrix multiplication is composition of transformations: \((\mA\mB)\vx=\mA(\mB\vx)\) means “apply \(\mB\), then \(\mA\).” That is why the order matters and why the inner dimensions must agree — the output of \(\mB\) must be a valid input to \(\mA\). A second useful reading: column \(j\) of \(\mA\mB\) is \(\mA\) applied to column \(j\) of \(\mB\).
\[\begin{bmatrix}1&2\\0&1\end{bmatrix}\!\begin{bmatrix}3&1\\2&4\end{bmatrix} =\begin{bmatrix}1\cdot3+2\cdot2 & 1\cdot1+2\cdot4\\ 0\cdot3+1\cdot2 & 0\cdot1+1\cdot4\end{bmatrix} =\begin{bmatrix}7&9\\2&4\end{bmatrix}.\]
Matrix multiplication is not commutative: in general \(\mA\mB\neq\mB\mA\) (often they do not even have compatible shapes). It is associative (\(\mA(\mB\mC)=(\mA\mB)\mC\)) and distributive. Also beware: \(\mA\mB=\mathbf{0}\) does not imply \(\mA=\mathbf{0}\) or \(\mB=\mathbf{0}\), and \(\mA\mB=\mA\mC\) does not imply \(\mB=\mC\) unless \(\mA\) is invertible.
A neural-network layer is one matrix multiply plus a bias and a nonlinearity: \(\vh=\phi(\mW\vx+\vb)\). A whole batch of inputs is processed at once as \(\mX\mW\tp\) — this is why GPUs, which are matrix-multiply engines, power deep learning. Stacking layers is composing linear maps separated by nonlinearities; without the nonlinearity the composition \(\mW_2\mW_1\) collapses to a single matrix. Essentially all training time is spent on matrix multiplications.
The transpose
The transpose \(\mA\tp\) flips rows and columns: \((\mA\tp)_{ij}=a_{ji}\). Key rules: \((\mA\tp)\tp=\mA\), \((\mA+\mB)\tp=\mA\tp+\mB\tp\), and the order-reversing \[(\mA\mB)\tp = \mB\tp\mA\tp.\] The combination \(\mA\tp\mA\) (always square, symmetric, and positive semidefinite) appears constantly — it is the matrix behind the normal equations and PCA.
Special matrices to recognize
| Type | Property | Why it matters |
|---|---|---|
| Identity \(\mI\) | \(\mI\vx=\vx\); ones on diagonal | the “do nothing” map |
| Diagonal \(\mD\) | off-diagonal zeros | scales each axis independently |
| Symmetric | \(\mA\tp=\mA\) | real eigenvalues; spectral theorem (Ch. [ch:symmetric]) |
| Triangular | zeros above/below diagonal | easy to solve; output of elimination |
| Orthogonal \(\mQ\) | \(\mQ\tp\mQ=\mI\) | rotations/reflections; preserve length |
| Permutation | rows of \(\mI\) reordered | row swaps in elimination |
Orthogonal matrices are the “rigid motions”: they preserve every length and angle because \(\norm{\mQ\vx}=\norm{\vx}\). They are numerically golden — they never amplify error — which is exactly why the QR and SVD factorizations (built from orthogonal matrices) are the stable backbone of numerical computing.
The inverse
A square matrix \(\mA\) is invertible (nonsingular) if there exists \(\mA\inv\) with \(\mA\inv\mA=\mA\mA\inv=\mI\). Then \(\mA\vx=\vb\) has the unique solution \(\vx=\mA\inv\vb\). Equivalent conditions: \(\mA\) has full rank, nonzero determinant, no zero eigenvalue, and trivial null space.
For \(2\times2\), \(\begin{psmallmatrix}a&b\\c&d\end{psmallmatrix}\inv=\frac{1}{ad-bc}\begin{psmallmatrix}d&-b\\-c&a\end{psmallmatrix}\), valid when \(ad-bc\neq0\). The order-reversing rule holds: \((\mA\mB)\inv=\mB\inv\mA\inv\).
In numerical code, do not compute an explicit inverse to solve a system. Use rather than : it is faster, more accurate, and avoids amplifying round-off. Forming \(\mA\inv\) is almost always a sign of a suboptimal algorithm. (Inverses are fine for derivations and for small fixed matrices.)
Block matrices and the LU factorization
Matrices can be partitioned into blocks and multiplied block-by-block (provided shapes conform) — a perspective that makes many proofs and efficient algorithms transparent.
Elimination itself is a factorization. Recording the multipliers used to zero out entries gives the LU decomposition: \[\mA = \mL\mU,\] with \(\mL\) lower-triangular (the multipliers) and \(\mU\) upper-triangular (the echelon form). With row swaps it becomes \(\mP\mA=\mL\mU\). LU is how production solvers handle \(\mA\vx=\vb\): factor once, then solve cheaply for many right-hand sides by forward/back substitution.
import numpy as np
from scipy.linalg import lu
A = np.array([[2.0,1,1],[4,3,3],[8,7,9]])
P, L, U = lu(A) # A = P @ L @ U
print(np.allclose(P @ L @ U, A)) # True
print(np.linalg.inv(A) @ A) # ~ identity (but prefer solve in practice)Chapter summary
A matrix is a table, a transformation, and a collection of vectors — switch views freely.
\(\mA\vx\) is a linear combination of the columns of \(\mA\); this is the key identity.
Matrix multiplication composes transformations: associative and distributive, but not commutative.
Transpose reverses order: \((\mA\mB)\tp=\mB\tp\mA\tp\); \(\mA\tp\mA\) is symmetric PSD and ubiquitous.
Know the special matrices (identity, diagonal, symmetric, orthogonal, triangular); orthogonal matrices preserve length.
The inverse undoes a map but is rarely computed numerically; elimination = the \(\mA=\mL\mU\) factorization.