Almost everyone who “bounces off” machine learning does so for the same reason: they tried to learn the algorithms before they had the language the algorithms are written in. That language is three branches of mathematics — linear algebra, calculus, and probability/statistics — plus a small, fixed toolbox of software. You do not need to become a mathematician. You need fluency: the ability to look at \(\nabla_\vw \, \tfrac{1}{2}\norm{\mX\vw-\vy}^2 = \mX\tp(\mX\vw-\vy)\) and read it as a sentence rather than a wall of symbols.

How to study Phase 0

Resist the urge to “complete” mathematics. Learn each concept to the depth where you can (a) explain it in words, (b) compute a small example by hand, and (c) reproduce it in NumPy. Use 3Blue1Brown for geometric intuition, then Mathematics for Machine Learning (free PDF) for rigor, then code. Spend roughly 40% linear algebra, 25% calculus, 25% probability/stats, 10% tooling.

Linear algebra

Vectors: the atoms

A vector \(\vx \in \R^n\) is an ordered list of \(n\) numbers. In ML it is almost always a data point (a row of features) or a set of parameters. Two complementary pictures matter:

• Geometric: an arrow from the origin to a point in \(n\)-dimensional space.

• Data: “this house = [1800 sq ft, 3 beds, 1995 built, …]”.

Norms: measuring size

A norm measures the length of a vector. The three you must know: \[\norm{\vx}_1 = \sum_i \abs{x_i}\ (\text{L1, ``Manhattan''}),\quad \norm{\vx}_2 = \sqrt{\textstyle\sum_i x_i^2}\ (\text{L2, ``Euclidean''}),\quad \norm{\vx}_\infty = \max_i \abs{x_i}.\] For matrices, the Frobenius norm \(\norm{\mA}_F = \sqrt{\sum_{i,j} A_{ij}^2}\) is the L2 norm of the flattened matrix. These reappear immediately in Phase 1 as regularizers: L2 (Ridge) shrinks weights smoothly, L1 (Lasso) drives some weights exactly to zero.

Matrices and matrix multiplication

A matrix \(\mA \in \R^{m\times n}\) is a grid of numbers, and is best understood as a linear transformation from \(\R^n\) to \(\R^m\). The product \(\mA\vx\) is a new vector; \((\mA\mB)\) composes two transformations.

Transpose, identity, inverse

The transpose \(\mA\tp\) swaps rows and columns: \((\mA\tp)_{ij}=A_{ji}\), with \((\mA\mB)\tp=\mB\tp\mA\tp\). The identity \(\mI\) has ones on the diagonal and acts as the “do nothing” transform: \(\mI\vx=\vx\). The inverse \(\mA^{-1}\) (when it exists, for square \(\mA\)) undoes the transform: \(\mA^{-1}\mA=\mI\).

Rank, determinant, and what can go wrong

The rank of a matrix is the number of linearly independent columns — the dimension of the space its outputs actually span. A matrix is full rank when no column is a combination of the others. The determinant \(\det(\mA)\) measures how the transform scales volume; \(\det(\mA)=0\) means the transform collapses space into a lower dimension (and the matrix is singular / non-invertible).

Eigenvalues, eigenvectors, and eigendecomposition

For a square matrix \(\mA\), a nonzero vector \(\vv\) is an eigenvector with eigenvalue \(\lambda\) if \[\mA\vv = \lambda\vv.\] The transform \(\mA\) acts on \(\vv\) by pure scaling — no rotation. Eigenvectors are the “natural axes” of the transformation.

Singular Value Decomposition (SVD)

SVD generalizes eigendecomposition to any matrix, square or not. Every \(\mA\in\R^{m\times n}\) factors as \[\mA = \mathbf{U}\boldsymbol{\Sigma}\mathbf{V}\tp,\] where \(\mathbf{U}\in\R^{m\times m}\) and \(\mathbf{V}\in\R^{n\times n}\) are orthogonal, and \(\boldsymbol{\Sigma}\) is diagonal with non-negative singular values \(\sigma_1\ge\sigma_2\ge\dots\ge 0\). Geometrically every linear map is a rotation, then a scaling, then another rotation.

import numpy as np
U, s, Vt = np.linalg.svd(A, full_matrices=False)   # A is (m, n)
k = 30
A_k = (U[:, :k] * s[:k]) @ Vt[:k, :]               # rank-k reconstruction
energy = s[:k].sum() / s.sum()                      # fraction of "energy" kept

Vector spaces, basis, and projection

A basis is a minimal set of vectors whose linear combinations generate a space; coordinates are just the coefficients in some basis. Projection onto a subspace finds the closest point in that subspace — the geometric heart of least-squares regression. The projection of \(\vy\) onto the column space of \(\mX\) is \(\hat\vy = \mX(\mX\tp\mX)^{-1}\mX\tp\vy\); the residual \(\vy-\hat\vy\) is orthogonal to that space. That orthogonality is the normal equation you will derive in Phase 1.

Calculus

Derivatives and the chain rule

The derivative \(f'(x)\) is the instantaneous rate of change — the slope of the tangent line. The chain rule composes rates of change: \[\frac{d}{dx} f(g(x)) = f'(g(x))\,g'(x).\]

Partial derivatives, gradients, Jacobians, Hessians

For a function \(f(\vx)\) of several variables, the partial derivative \(\partial f/\partial x_i\) varies one input while holding the rest fixed. Collecting them gives the gradient: \[\nabla f(\vx) = \Big(\tfrac{\partial f}{\partial x_1}, \dots, \tfrac{\partial f}{\partial x_n}\Big)\tp \in \R^n.\]

For a vector-valued function \(\mathbf{f}:\R^n\to\R^m\), the Jacobian \(\mathbf{J}\in\R^{m\times n}\) collects all partials \(J_{ij}=\partial f_i/\partial x_j\). The Hessian \(\mathbf{H}\in\R^{n\times n}\) collects second partials \(H_{ij}=\partial^2 f/\partial x_i\partial x_j\) and describes curvature.

Gradients you will use constantly

Memorize these matrix-calculus identities; they cover most of classical ML. \[\begin{align*} \nabla_\vx (\va\tp\vx) &= \va, & \nabla_\vx (\vx\tp\mA\vx) &= (\mA+\mA\tp)\vx, \\ \nabla_\vx \tfrac{1}{2}\norm{\vx}_2^2 &= \vx, & \nabla_\vw \tfrac{1}{2}\norm{\mX\vw-\vy}_2^2 &= \mX\tp(\mX\vw-\vy). \end{align*}\]

Why gradient descent works

Gradient descent iterates \(\vw_{t+1}=\vw_t-\eta\,\nabla f(\vw_t)\). The first-order Taylor expansion explains it: near \(\vw_t\), \(f(\vw_t+\Delta)\approx f(\vw_t)+\nabla f(\vw_t)\tp\Delta\). Choosing \(\Delta=-\eta\nabla f\) makes the inner product as negative as possible (for small step), so \(f\) decreases. The step size \(\eta\) (learning rate) trades speed against the risk of overshooting — too large and you diverge, too small and you crawl.

Probability & statistics

Random variables and distributions

A random variable maps outcomes to numbers. You must know these distributions and where each shows up:

Expectation, variance, covariance, correlation

\[\E[X]=\sum_x x\,p(x)\ \text{(or }\textstyle\int x\,p(x)\,dx),\quad \Var(X)=\E[(X-\E X)^2]=\E[X^2]-(\E X)^2.\] For two variables, covariance \(\Cov(X,Y)=\E[(X-\E X)(Y-\E Y)]\) measures co-movement, and correlation \(\rho=\Cov(X,Y)/(\sigma_X\sigma_Y)\in[-1,1]\) is its scale-free version. The covariance matrix \(\mS\) of a random vector collects all pairwise covariances; it is symmetric PSD, and its eigen-structure is PCA.

Conditional probability and Bayes’ theorem

\[\Prob(A\mid B)=\frac{\Prob(A\cap B)}{\Prob(B)},\qquad \underbrace{\Prob(\theta\mid \text{data})}_{\text{posterior}}=\frac{\overbrace{\Prob(\text{data}\mid\theta)}^{\text{likelihood}}\,\overbrace{\Prob(\theta)}^{\text{prior}}}{\underbrace{\Prob(\text{data})}_{\text{evidence}}}.\]

MLE and MAP estimation

Given data and a parametric model, Maximum Likelihood Estimation (MLE) chooses parameters that make the observed data most probable: \[\hat{\vtheta}_{\text{MLE}}=\argmax_{\vtheta} \prod_i p(x_i\mid\vtheta)=\argmax_{\vtheta} \sum_i \log p(x_i\mid\vtheta).\] Maximum A Posteriori (MAP) adds a prior: \(\hat{\vtheta}_{\text{MAP}}=\argmax_{\vtheta} \big[\sum_i\log p(x_i\mid\vtheta)+\log p(\vtheta)\big]\).

Inferential statistics: CLT, intervals, hypothesis tests

The Central Limit Theorem states that the mean of many independent samples is approximately Gaussian, regardless of the underlying distribution. This is why averages are well-behaved and why error bars work. A confidence interval quantifies uncertainty in an estimate; a \(p\)-value is the probability of seeing data at least as extreme as observed if the null hypothesis were true.

Tooling: the scientific Python stack

You already program well, so this is about idioms, not syntax. The goal is to think in vectorized array operations, not Python loops.

NumPy — the foundation

NumPy’s is the substrate everything else builds on. The two ideas that separate beginners from fluent users are vectorization and broadcasting.

import numpy as np
X = np.random.randn(1000, 5)          # 1000 samples, 5 features
mu, sigma = X.mean(axis=0), X.std(axis=0)   # shape (5,)
Xz = (X - mu) / sigma                  # broadcast (1000,5)-(5,) -> (1000,5)

# Vectorized vs. loop: compute pairwise squared distances to a centroid
c = X.mean(0)
d2 = ((X - c) ** 2).sum(axis=1)        # shape (1000,), no Python loop

Pandas — tabular data

Pandas s handle real-world messy tables: mixed types, missing values, joins, group-bys. Learn , indexing with /, , , and handling (, ). You will live in Pandas during the Phase 1 EDA and feature-engineering work.

Matplotlib & Seaborn — seeing the data

Plotting is not decoration; it is debugging. Always look at your data and your model’s errors. Histograms reveal skew and outliers, scatter plots reveal relationships, residual plots reveal model misfit, and learning curves reveal over/underfitting (Phase 1). Seaborn sits on Matplotlib for fast statistical plots (, of a correlation matrix).

Jupyter / Colab, Git, and a little SQL

• Jupyter / Colab: interactive exploration. Colab gives free GPUs (vital from Phase 3). Beware hidden state from out-of-order cell execution — restart-and-run-all before trusting a notebook.

• Git: version every experiment. Commit notebooks and configs; keep large data and model weights out of the repo (use , DVC, or a model registry later in Phase 7).

• SQL: you are likely already strong here. For ML it is the data-wrangling front door — , , , window functions for feature aggregation pull training sets straight from warehouses.

Checkpoint project

import numpy as np

def gradient_descent(grad_fn, w0, lr=0.1, steps=100):
    """Generic GD; grad_fn(w) returns the gradient at w."""
    w = np.array(w0, dtype=float)
    history = [w.copy()]
    for _ in range(steps):
        w = w - lr * grad_fn(w)
        history.append(w.copy())
    return w, np.array(history)

# Part C: linear regression via GD, then check vs. the normal equation.
rng = np.random.default_rng(0)
X = np.c_[np.ones(200), rng.uniform(-3, 3, 200)]   # design matrix (200,2)
w_true = np.array([1.5, -2.0])
y = X @ w_true + 0.5 * rng.standard_normal(200)

grad = lambda w: X.T @ (X @ w - y) / len(y)         # MSE gradient
w_gd, hist = gradient_descent(grad, [0.0, 0.0], lr=0.05, steps=500)
w_closed = np.linalg.solve(X.T @ X, X.T @ y)         # normal equation
print("GD     :", w_gd)
print("closed :", w_closed)        # should match to a few decimals

Phase 0 self-check

Before moving to Phase 1, you should be able to, without notes:

• Explain what \(\mA\vx\) does geometrically and compute a small matrix product by hand.

• State what eigenvectors/eigenvalues and the SVD are, and name one ML use of each.

• Derive the gradient of the least-squares loss and the normal equation.

• Explain why \(-\nabla f\) is the descent direction and what the learning rate controls.

• Recover MSE and cross-entropy as negative log-likelihoods, and L2/L1 as MAP priors.

• Apply Bayes’ theorem to a base-rate problem and interpret the result.

• Vectorize a computation with NumPy broadcasting instead of a Python loop.