Eigenvalues reveal the hidden axes of a transformation — the special directions a matrix merely stretches without rotating. They explain the long-term behavior of dynamical systems, the principal directions of data, the curvature of loss surfaces, and the ranking behind PageRank. This chapter develops eigenvalues, eigenvectors, and diagonalization; the next specializes to the beautifully behaved symmetric case.
The eigenvalue equation
For a square matrix \(\mA\in\R^{n\times n}\), a nonzero vector \(\vv\) is an eigenvector with eigenvalue \(\lambda\) if \[\mA\vv = \lambda\vv.\] Along \(\vv\), the transformation acts as pure scaling by \(\lambda\) — no change of direction (unless \(\lambda<0\), which flips it).
Most vectors are knocked off their line when you apply \(\mA\). Eigenvectors are the rare directions that stay on their own line; the eigenvalue says by how much they stretch. These special directions are the “natural axes” of the matrix — in the eigenvector basis, the tangled action of \(\mA\) becomes a simple set of independent scalings. That is the whole motivation for diagonalization.
Finding eigenvalues: the characteristic polynomial
Rewrite \(\mA\vv=\lambda\vv\) as \((\mA-\lambda\mI)\vv=\vzero\). A nonzero \(\vv\) exists exactly when \(\mA-\lambda\mI\) is singular:
The eigenvalues of \(\mA\) are the roots of the characteristic polynomial \[\det(\mA-\lambda\mI)=0.\] For each eigenvalue \(\lambda\), the corresponding eigenvectors span the eigenspace \(\nullsp(\mA-\lambda\mI)\).
For \(\mA=\begin{bmatrix}2&1\\1&2\end{bmatrix}\), \(\det(\mA-\lambda\mI)=(2-\lambda)^2-1=\lambda^2-4\lambda+3=(\lambda-1)(\lambda-3)\). So \(\lambda_1=3\) with eigenvector \([1,1]\tp\) and \(\lambda_2=1\) with eigenvector \([1,-1]\tp\). The matrix stretches by \(3\) along the diagonal and by \(1\) perpendicular to it.
The sum of the eigenvalues equals the trace, \(\sum_i\lambda_i=\trace(\mA)\), and their product equals the determinant, \(\prod_i\lambda_i=\det(\mA)\). Hence \(\mA\) is invertible iff no eigenvalue is zero.
Diagonalization
If \(\mA\in\R^{n\times n}\) has \(n\) linearly independent eigenvectors, gather them as the columns of \(\mP\) and the eigenvalues into a diagonal \(\mD\). Then \[\mA = \mP\mD\mP\inv,\qquad \mD=\diag(\lambda_1,\dots,\lambda_n).\] \(\mA\) is then called diagonalizable: in the eigenvector basis it is just the diagonal scaling \(\mD\).
Diagonalization makes matrix powers trivial: \(\mA^k=\mP\mD^k\mP\inv\), and \(\mD^k\) just raises each eigenvalue to the \(k\). This is why eigenvalues control long-term behavior. Powering a matrix repeatedly (a Markov chain, a recurrent network, an iterative solver) amplifies directions with \(|\lambda|>1\) and kills those with \(|\lambda|<1\); the largest-magnitude eigenvalue eventually dominates.
A system \(\vx_{t+1}=\mA\vx_t\) evolves as \(\vx_t=\mA^t\vx_0=\mP\mD^t\mP\inv\vx_0=\sum_i c_i\lambda_i^t\vv_i\). If one \(|\lambda_i|>1\), that mode blows up; if all \(|\lambda_i|<1\), the system decays to zero; the steady state corresponds to \(\lambda=1\).
Not every matrix is diagonalizable. Defective matrices (e.g. \(\begin{psmallmatrix}1&1\\0&1\end{psmallmatrix}\)) lack a full set of independent eigenvectors. Also, real matrices can have complex eigenvalues (rotations: \(\begin{psmallmatrix}0&-1\\1&0\end{psmallmatrix}\) has eigenvalues \(\pm i\)). Both problems vanish for symmetric matrices (Chapter [ch:symmetric]) — which is one reason symmetric matrices are so beloved in ML. When diagonalization fails, the SVD (Chapter [ch:svd]) always works.
Applications: Markov chains and PageRank
A Markov matrix (nonnegative columns summing to \(1\)) always has \(\lambda=1\) as its largest eigenvalue; the corresponding eigenvector is the stationary distribution the chain converges to. Google’s original PageRank is exactly this: model a random web surfer as a Markov chain over links, and rank pages by the stationary distribution — the dominant eigenvector of the link matrix, found by the power method (repeated multiplication).
Eigen-thinking is everywhere in ML. The eigenvectors of a covariance matrix are the principal components, and the eigenvalues are the variances along them (PCA, Chapter [ch:symmetric]). The eigenvalues of the Hessian of a loss describe its curvature: their ratio (the condition number) sets how badly gradient descent zig-zags and how small the learning rate must be. Spectral clustering and graph neural networks use eigenvectors of the graph Laplacian. PageRank and many recommendation/ranking methods are dominant-eigenvector computations.
import numpy as np
A = np.array([[2.,1],[1,2]])
vals, vecs = np.linalg.eig(A) # eigenvalues, eigenvectors (columns)
print(vals) # [3., 1.]
# Reconstruct A = P D P^{-1}
P, D = vecs, np.diag(vals)
print(np.allclose(P @ D @ np.linalg.inv(P), A)) # True
# Power method for the dominant eigenvector:
v = np.random.randn(2)
for _ in range(100): v = A @ v; v /= np.linalg.norm(v)
print(v) # aligns with eigenvector for lambda=3Chapter summary
Eigenvectors are directions scaled (not rotated) by \(\mA\); eigenvalues are the scale factors: \(\mA\vv=\lambda\vv\).
Eigenvalues are roots of \(\det(\mA-\lambda\mI)=0\); \(\sum\lambda_i=\trace\), \(\prod\lambda_i=\det\).
Diagonalization \(\mA=\mP\mD\mP\inv\) makes powers and dynamics easy; the dominant eigenvalue governs long-term behavior.
Not all matrices diagonalize (defective or complex eigenvalues) — but symmetric ones always do, and the SVD always exists.
Eigen-methods power PCA, Hessian/curvature analysis, spectral clustering, and PageRank.