Notation and Formula Reference

A compact reference for the symbols and identities used throughout the book.

Notation

Core identities

Products and transposes. \[(\mA\mB)\tp=\mB\tp\mA\tp,\quad (\mA\mB)\inv=\mB\inv\mA\inv,\quad (\mA\tp)\inv=(\mA\inv)\tp.\] Determinant and trace. \[\det(\mA\mB)=\det\mA\det\mB,\quad \det(\mA\tp)=\det\mA,\quad \trace(\mA\mB)=\trace(\mB\mA),\quad \det\mA=\prod_i\lambda_i,\quad \trace\mA=\sum_i\lambda_i.\] Rank–nullity. \(\ \rank(\mA)+\dimn\nullsp(\mA)=n.\)

Factorizations. \[\mA=\mL\mU,\quad \mA=\mQ\mR,\quad \mA=\mL\mL\tp\ (\text{SPD}),\quad \mA=\mP\mD\mP\inv,\quad \mA=\mQ\mLam\mQ\tp\ (\text{sym.}),\quad \mA=\mU\mSig\mV\tp.\] Least squares & projection. \[\mA\tp\mA\,\hat\vx=\mA\tp\vb,\qquad \mP=\mA(\mA\tp\mA)\inv\mA\tp,\qquad \hat\vx=\mA^{+}\vb.\] Low-rank (Eckart–Young). \[\mA_k=\sum_{i=1}^k\sigma_i\vu_i\vv_i\tp,\qquad \norm{\mA-\mA_k}_2=\sigma_{k+1}.\] Matrix calculus. \[\nabla_{\vx}(\va\tp\vx)=\va,\quad \nabla_{\vx}(\vx\tp\mA\vx)=(\mA+\mA\tp)\vx,\quad \nabla_{\vx}\norm{\mA\vx-\vb}_2^2=2\mA\tp(\mA\vx-\vb).\]

Equivalent conditions for an invertible \(n\times n\) matrix

The following are all equivalent (the “invertible matrix theorem”): \(\mA\) is invertible; \(\det\mA\neq0\); \(\rank\mA=n\); columns (and rows) are linearly independent; \(\nullsp(\mA)=\{\vzero\}\); \(\mA\vx=\vb\) has a unique solution for every \(\vb\); \(0\) is not an eigenvalue; all singular values are positive; the columns are a basis of \(\R^n\).

Greek letters used

\(\alpha,\beta,\gamma\) (scalars/coefficients), \(\theta\) (angle, parameters \(\vtheta\)), \(\lambda\) (eigenvalue, regularization), \(\sigma\) (singular value, std. dev.), \(\mu\) (mean), \(\Sigma\) (covariance, sum), \(\Lambda\) (eigenvalue matrix), \(\phi\) (activation), \(\eta\) (learning rate).

Minimal NumPy / PyTorch reference