We have treated matrices as arrays of numbers. This chapter elevates them to their true identity: matrices are linear transformations once a basis is fixed. This view explains why matrix multiplication is defined the way it is, what “similar matrices” means, and how changing coordinates can dramatically simplify a problem — the idea that culminates in diagonalization and the SVD.

Linear transformations

A function \(T:\R^n\to\R^m\) is a linear transformation if it respects linear combinations: \[T(\vu+\vv)=T(\vu)+T(\vv)\quad\text{and}\quad T(c\vv)=c\,T(\vv)\qquad\text{for all }\vu,\vv,c.\] Equivalently, \(T(c\vu+d\vv)=c\,T(\vu)+d\,T(\vv)\). A linear map always sends \(\vzero\) to \(\vzero\) and sends lines to lines.

If \(T:\R^n\to\R^m\) is linear, then \(T(\vx)=\mA\vx\) where the \(j\)-th column of \(\mA\) is \(T(\ve_j)\) — the image of the \(j\)-th standard basis vector. Conversely, every matrix defines a linear map.

This is profound and practical: to know a linear transformation, you only need to know what it does to the basis vectors. Their images become the columns of the matrix, and linearity fills in everything else. Because \(T(\sum_j x_j\ve_j)=\sum_j x_j T(\ve_j)\), the output is the same linear combination of the columns — which is exactly the column view of \(\mA\vx\).

\[\text{rotate by }\theta:\begin{bmatrix}\cos\theta&-\sin\theta\\ \sin\theta&\cos\theta\end{bmatrix},\quad \text{scale: }\begin{bmatrix}s_x&0\\0&s_y\end{bmatrix},\quad \text{shear: }\begin{bmatrix}1&k\\0&1\end{bmatrix},\quad \text{project onto }x:\begin{bmatrix}1&0\\0&0\end{bmatrix}.\] Each column tells you where \(\ve_1\) and \(\ve_2\) land. The projection has determinant \(0\) — it collapses the plane onto a line (a nontrivial null space).

Kernel and image

The kernel (null space) of \(T\) is \(\{\vx:T(\vx)=\vzero\}\); the image (range, column space) is \(\{T(\vx)\}\). These are exactly two of the four fundamental subspaces, now wearing their transformation hats. \(T\) is one-to-one iff its kernel is trivial, and onto iff its image is all of \(\R^m\).

Composition is multiplication

If \(S(\vx)=\mB\vx\) then \(T(\vx)=\mA\vx\), the composition \((T\circ S)(\vx)=T(S(\vx))=\mA\mB\vx\). So composing transformations corresponds to multiplying their matrices — and this is precisely why matrix multiplication is defined by the row-times-column rule. The non-commutativity of multiplication is just the fact that “rotate then shear” differs from “shear then rotate.”

Change of basis

The same vector has different coordinates in different bases, and the same transformation has different matrices. Choosing the right basis can turn a messy matrix into a simple one.

Let the columns of an invertible matrix \(\mP\) be a new basis (written in the standard basis). A vector with standard coordinates \(\vx\) has new coordinates \(\vx' = \mP\inv\vx\), and conversely \(\vx=\mP\vx'\).

Two square matrices \(\mA\) and \(\mB\) are similar if \(\mB=\mP\inv\mA\mP\) for some invertible \(\mP\). Similar matrices represent the same linear transformation in different bases. They share determinant, trace, rank, and eigenvalues.

Change of basis is the strategic heart of advanced linear algebra. “Diagonalization” (Chapter [ch:eigen]) means: find a basis of eigenvectors in which the transformation is just independent scalings — \(\mA=\mP\mD\mP\inv\). The SVD (Chapter [ch:svd]) finds two orthonormal bases (input and output) in which any matrix becomes diagonal. Almost every powerful factorization is the sentence “in the right coordinates, this map is simple.”

Change of basis is what PCA does: rotate the data into the coordinate system of its principal axes, where the covariance is diagonal and the variance is concentrated in the first few coordinates. A neural network can be read as a learned sequence of changes of representation (“basis”), each layer re-expressing the data in coordinates where the next task is easier — ultimately a basis where the classes are linearly separable. The recurring engineering move “project into a better space, act, project back” is change of basis.

import numpy as np
theta = np.pi/4
R = np.array([[np.cos(theta), -np.sin(theta)],
              [np.sin(theta),  np.cos(theta)]])   # rotation
print(R @ np.array([1,0]))         # where e1 lands = first column of R
# Similarity: same map, new basis P
P = np.array([[1.,1],[0,1]])
A = np.array([[2.,0],[0,3]])
B = np.linalg.inv(P) @ A @ P        # similar to A: same eigenvalues 2,3
print(np.linalg.eigvals(B))

Chapter summary

  • A linear map preserves linear combinations; it is fully determined by what it does to a basis.

  • The columns of its matrix are the images of the basis vectors \(T(\ve_j)\).

  • Composition of maps \(=\) multiplication of matrices (hence the multiplication rule and its non-commutativity).

  • Change of basis re-expresses vectors and maps; similar matrices (\(\mB=\mP\inv\mA\mP\)) are one map in different coordinates.

  • “In the right basis, the map is simple” previews diagonalization and the SVD.