This is the conceptual heart of linear algebra. Everything so far — vectors, systems, matrices — now gets organized by a single powerful abstraction: the vector space and its subspaces. The payoff is the Fundamental Theorem of Linear Algebra, which describes the four subspaces hidden inside every matrix and how they fit together. Mastering this chapter is what separates “I can do the arithmetic” from “I understand what is going on.”
Vector spaces and subspaces
A vector space is a set \(V\) closed under addition and scalar multiplication, obeying the usual algebraic rules (associativity, distributivity, a zero vector, additive inverses). The prototype is \(\R^n\), but matrices, polynomials, and functions also form vector spaces.
A subspace of \(V\) is a nonempty subset \(W\subseteq V\) that is itself a vector space: it contains \(\vzero\) and is closed under linear combinations (if \(\vu,\vw\in W\) then \(c\vu+d\vw\in W\)). Geometrically, subspaces of \(\R^n\) are lines, planes, and hyperplanes through the origin.
A line or plane that does not pass through the origin is not a subspace — it fails to contain \(\vzero\) and is not closed under scaling. (Such “shifted” sets are affine sets.) Always check the origin first.
Span, independence, basis, dimension
The span of vectors \(\vv_1,\dots,\vv_k\) is the set of all their linear combinations — the smallest subspace containing them. The vectors are linearly independent if no one of them is a linear combination of the others, equivalently if \(c_1\vv_1+\cdots+c_k\vv_k=\vzero\) forces all \(c_i=0\).
A basis of a subspace is a linearly independent set that spans it. Every basis of a given subspace has the same number of vectors; that number is the dimension. A basis gives every vector unique coordinates.
A basis is a minimal, non-redundant coordinate system: enough vectors to reach everything (spanning), with none wasted (independent). “Dimension” is the number of independent directions — the true number of degrees of freedom, regardless of how the space is presented. In data terms, the dimension of the span of your features is how much non-redundant information they carry.
In \(\R^3\), the vectors \([1,0,0]\tp,[0,1,0]\tp,[1,1,0]\tp\) are dependent (the third is the sum of the first two) and span only the \(xy\)-plane (dimension 2). Dropping the redundant third vector leaves a basis for that plane.
The four fundamental subspaces
Every matrix \(\mA\in\R^{m\times n}\) gives rise to four subspaces. They are the key to understanding what \(\mA\) does and when \(\mA\vx=\vb\) is solvable.
For \(\mA\in\R^{m\times n}\) with rank \(r\):
Column space \(\col(\mA)\subseteq\R^m\): all linear combinations of the columns; equivalently all outputs \(\mA\vx\). Dimension \(r\).
Null space \(\nullsp(\mA)\subseteq\R^n\): all solutions of \(\mA\vx=\vzero\). Dimension \(n-r\).
Row space \(\col(\mA\tp)\subseteq\R^n\): all combinations of the rows. Dimension \(r\).
Left null space \(\nullsp(\mA\tp)\subseteq\R^m\): all \(\vy\) with \(\mA\tp\vy=\vzero\). Dimension \(m-r\).
The rank \(r\) is the number of pivots in the row echelon form — equivalently the dimension of the column space, and (remarkably) also of the row space. The number of independent columns equals the number of independent rows.
The rank–nullity theorem
For \(\mA\in\R^{m\times n}\), \[\rank(\mA) + \dimn\,\nullsp(\mA) = n.\] Each of the \(n\) columns either supports a pivot (contributing to rank) or a free variable (contributing to the null space). There is no third option, so the two counts must add to \(n\).
The Fundamental Theorem of Linear Algebra
In \(\R^n\), the row space and the null space are orthogonal complements: every row-space vector is orthogonal to every null-space vector, and together they fill \(\R^n\). In \(\R^m\), the column space and the left null space are orthogonal complements and together fill \(\R^m\).
This single picture explains a matrix completely. \(\mA\) sends the row space bijectively onto the column space (the “real action”), and crushes the null space to zero. The left null space is the part of the output space that \(\mA\) can never reach. Solvability of \(\mA\vx=\vb\) is now obvious: it is solvable iff \(\vb\) lies in the column space, and the solution is unique iff the null space is trivial.
Let \(\mA=\begin{bmatrix}1&2&3\\2&4&6\end{bmatrix}\). Row reduction gives \(\begin{bmatrix}1&2&3\\0&0&0\end{bmatrix}\), one pivot, so \(r=1\). The column space is the line through \([1,2]\tp\) (all columns are multiples of it). The null space (\(n-r=2\) dimensional) is spanned by \([-2,1,0]\tp\) and \([-3,0,1]\tp\) (from the free variables). The row space is the line through \([1,2,3]\tp\), orthogonal to that null space.
Why the null space governs regularization
If a data matrix \(\mX\) has a nontrivial null space — which happens whenever there are more features than samples, or features are collinear — then \(\mX\vw=\vy\) has infinitely many equally good solutions: add any null-space vector to \(\vw\) and predictions on the training data are unchanged. The model is unidentifiable. Regularization (Ridge/Lasso) breaks the tie by selecting the minimum-norm solution from this infinite family; it is not a mere nicety but a structural necessity. The rank of \(\mX\) is the intrinsic number of independent features, and the null space is precisely the directions in which the data tells you nothing.
import numpy as np
from scipy.linalg import null_space
A = np.array([[1.,2,3],[2,4,6]])
print(np.linalg.matrix_rank(A)) # 1
print(null_space(A)) # basis for the 2-D null space
# Column space basis: independent columns (here just the first column).Chapter summary
Subspaces are lines/planes through the origin, closed under linear combinations.
Span, independence, basis, dimension formalize “non-redundant coordinate system.”
Every matrix has four fundamental subspaces; rank \(r\) governs all their dimensions.
Rank–nullity: \(\rank + \dim(\text{null}) = n\).
Row space \(\perp\) null space (in \(\R^n\)); column space \(\perp\) left null space (in \(\R^m\)).
A nontrivial null space \(\Rightarrow\) non-unique solutions \(\Rightarrow\) regularization is required.