Symmetric matrices (\(\mA\tp=\mA\)) are the best-behaved objects in linear algebra, and — not coincidentally — the ones that appear most in machine learning: covariance matrices, Gram matrices, kernel matrices, and Hessians are all symmetric. They have real eigenvalues, orthogonal eigenvectors, and an especially clean diagonalization (the spectral theorem). This chapter also introduces quadratic forms and positive definiteness, the language of optimization landscapes.

The spectral theorem

Every real symmetric matrix \(\mA\) can be diagonalized by an orthogonal matrix: \[\mA = \mQ\mLam\mQ\tp,\qquad \mQ\tp\mQ=\mI,\ \ \mLam=\diag(\lambda_1,\dots,\lambda_n).\] All eigenvalues \(\lambda_i\) are real, and the eigenvectors (columns of \(\mQ\)) can be chosen orthonormal. Equivalently \(\mA=\sum_i \lambda_i\,\vq_i\vq_i\tp\), a sum of scaled orthogonal projections.

The spectral theorem says a symmetric matrix is, in the right orthonormal coordinates, just a set of independent stretches along perpendicular axes — no shear, no rotation, no complex numbers. Geometrically it maps the unit sphere to an ellipsoid whose axes are the eigenvectors and whose semi-axis lengths are the eigenvalues. This is the cleanest possible structure, and it is why ML leans so heavily on symmetric matrices.

Quadratic forms

A quadratic form is a function \(Q(\vx)=\vx\tp\mA\vx\) with \(\mA\) symmetric — a pure “second-degree” function of the coordinates, with no linear or constant term. Examples: \(\norm{\vx}^2=\vx\tp\mI\vx\), and the exponent of a multivariate Gaussian.

Using the spectral theorem and the substitution \(\vy=\mQ\tp\vx\), \[Q(\vx)=\vx\tp\mA\vx=\sum_i \lambda_i\,y_i^2.\] So in the eigenvector coordinates a quadratic form is just a weighted sum of squares — the eigenvalues are the weights, and their signs determine its shape.

Positive definiteness

A symmetric matrix \(\mA\) is

  • positive definite if \(\vx\tp\mA\vx>0\) for all \(\vx\neq\vzero\) (all \(\lambda_i>0\));

  • positive semidefinite (PSD) if \(\vx\tp\mA\vx\ge0\) (all \(\lambda_i\ge0\));

  • indefinite if it takes both signs (mixed-sign eigenvalues).

Positive definiteness is the matrix version of “positive number,” and it is the multivariable test for a minimum. A quadratic form is a bowl (unique minimum) iff its matrix is positive definite, a dome (maximum) iff negative definite, and a saddle iff indefinite. For a loss function, the Hessian’s definiteness at a critical point tells you whether it is a minimum, maximum, or saddle — the central question of optimization.

For symmetric \(\mA\), the following are equivalent: (1) all eigenvalues positive; (2) all leading principal minors positive (Sylvester’s criterion); (3) all pivots positive in elimination; (4) \(\mA=\mB\tp\mB\) for some \(\mB\) with independent columns. Test (4) is why \(\mA\tp\mA\) and covariance matrices are always at least PSD.

The Rayleigh quotient

For symmetric \(\mA\), the Rayleigh quotient \(R(\vx)=\dfrac{\vx\tp\mA\vx}{\vx\tp\vx}\) is bounded by the extreme eigenvalues: \[\lambda_{\min}\le R(\vx)\le \lambda_{\max},\] with the maximum attained at the top eigenvector. This variational characterization — “the top eigenvector maximizes \(\vx\tp\mA\vx\) on the unit sphere” — is exactly the optimization PCA solves, and it underlies the power method and many spectral algorithms.

Covariance, Gram, and kernel matrices

The symmetric matrices of ML are all here. The covariance matrix \(\mSig=\frac1n\mX\tp\mX\) (for centered data) is symmetric PSD; its spectral decomposition is PCA — eigenvectors are principal directions, eigenvalues are variances, and the Rayleigh-quotient view says the first PC maximizes explained variance. Gram matrices \(\mX\mX\tp\) and kernel matrices \(K_{ij}=k(\vx_i,\vx_j)\) are symmetric PSD and power kernel methods, SVMs, and Gaussian processes. The Hessian of a loss is symmetric; its eigenvalues (curvature) determine convergence speed and whether a critical point is a minimum or a saddle — the latter being the norm in high-dimensional deep learning.

import numpy as np
X = np.random.randn(200, 3) @ np.array([[3,1,0],[0,2,0],[0,0,0.5]])
Xc = X - X.mean(0)
C = (Xc.T @ Xc) / len(Xc)              # symmetric PSD covariance
vals, vecs = np.linalg.eigh(C)         # eigh: for symmetric matrices (real, sorted)
print(vals[::-1])                      # variances along principal components
# PCA: project onto top-2 eigenvectors (largest eigenvalues)
PCs = vecs[:, ::-1][:, :2]
Z = Xc @ PCs

For symmetric matrices use , not : exploits symmetry to return real, sorted eigenvalues with orthonormal eigenvectors, and is faster and more accurate. Using the general can return tiny spurious imaginary parts from round-off.

Chapter summary

  • Spectral theorem: symmetric \(\mA=\mQ\mLam\mQ\tp\) with real eigenvalues and orthonormal eigenvectors.

  • A quadratic form \(\vx\tp\mA\vx=\sum_i\lambda_i y_i^2\) in eigen-coordinates; eigenvalue signs set its shape.

  • Positive definite \(\Leftrightarrow\) all \(\lambda_i>0\) \(\Leftrightarrow\) a bowl with a unique minimum; the Hessian test for optimization.

  • The Rayleigh quotient’s maximum is the top eigenvector — the optimization behind PCA.

  • Covariance, Gram, kernel, and Hessian matrices are all symmetric (P)SD — the workhorses of ML.