A neural network is a stack of learned feature transformations followed by a linear model. Each layer applies an affine map then a nonlinearity; composing many of them lets the network represent functions that no single linear model can. Everything you learned in Phases 0–1 carries over: dot products, gradients, cross-entropy, regularization, the bias–variance tradeoff. The new ingredient is depth, and the new skill is computing gradients through many composed functions: backpropagation.

Neural network basics

The perceptron and the MLP

A single artificial neuron computes \(a=\phi(\vw\tp\vx+b)\) — a dot product, a bias, and a nonlinearity \(\phi\). A multilayer perceptron (MLP) stacks layers of these: \[\vh^{(1)}=\phi(\mW^{(1)}\vx+\vb^{(1)}),\quad \vh^{(2)}=\phi(\mW^{(2)}\vh^{(1)}+\vb^{(2)}),\quad\dots,\quad \hat\vy=\mW^{(L)}\vh^{(L-1)}+\vb^{(L)}.\]

Activation functions

Loss functions

Regression uses MSE; classification uses cross-entropy on top of a softmax. Both are negative log-likelihoods (Phase 0). For numerical stability, frameworks fuse softmax and cross-entropy into one op ( takes raw logits, not probabilities).

Backpropagation — derive it once by hand

Backprop is the chain rule applied systematically over a computational graph, reusing intermediate results so the cost of all gradients equals a constant multiple of one forward pass.

Weight initialization

If weights are too large, activations explode; too small, they vanish. Principled schemes keep the variance of activations (and gradients) roughly constant across layers:

• Xavier/Glorot (for tanh/sigmoid): \(\Var(w)=\frac{2}{n_\text{in}+n_\text{out}}\).

• He (for ReLU): \(\Var(w)=\frac{2}{n_\text{in}}\) — accounts for ReLU zeroing half the inputs.

Optimization

Gradient descent variants

Batch GD uses the whole dataset per step (accurate, slow, memory-heavy). Stochastic GD (SGD) uses one example (noisy, fast). Mini-batch (32–512 examples) is the practical sweet spot — it exploits vectorized hardware and the gradient noise actually helps generalization.

Momentum and adaptive methods

\[\begin{align*} \text{Momentum:}&\quad \vv_t=\beta\vv_{t-1}+\nabla L,\quad \vtheta_t=\vtheta_{t-1}-\eta\vv_t \\ \text{RMSProp:}&\quad s_t=\rho s_{t-1}+(1-\rho)(\nabla L)^2,\quad \vtheta_t=\vtheta_{t-1}-\eta\,\nabla L/\sqrt{s_t+\epsilon} \\ \text{Adam:}&\quad \text{momentum} + \text{RMSProp, with bias correction} \end{align*}\]

Learning-rate scheduling

The single most important hyperparameter is the learning rate. Common schedules: step decay (cut by 10\(\times\) at milestones), cosine annealing (smooth decay to near zero), and warmup (ramp up over the first few hundred steps — essential for Transformers to avoid early divergence). A learning-rate finder (sweep LR and watch loss) quickly brackets a good value.

Vanishing/exploding gradients and clipping

In deep or recurrent nets, repeated multiplication by Jacobians can shrink gradients to zero (vanishing) or blow them up (exploding). Mitigations: ReLU-family activations, careful init, normalization layers, residual connections (Phase 3), and gradient clipping (cap the gradient norm) — standard practice when training RNNs and Transformers.

Regularization & generalization

Deep nets have enormous capacity, so controlling overfitting is central.

• Dropout: randomly zero a fraction of activations during training; an implicit ensemble that prevents co-adaptation. Disabled at inference.

• Weight decay (L2): penalize large weights; in AdamW this is decoupled from the adaptive scaling.

• Early stopping: halt when validation loss stops improving — cheap and effective.

• Data augmentation: expand the effective dataset with label-preserving transforms (crucial in vision, Phase 3).

Normalization layers

Batch normalization normalizes each feature across the mini-batch, stabilizing and accelerating training (and acting as a mild regularizer). Layer normalization normalizes across features within each example — batch-size independent and the standard in Transformers and RNNs.

Frameworks

Autograd and computational graphs

Frameworks record operations into a graph and apply backprop automatically. PyTorch builds the graph dynamically (define-by-run), so models are plain Python — easy to debug and to write custom architectures. Each tensor carries ; calling populates .

PyTorch — the recommended primary framework

PyTorch is transparent, dominant in research, and the path of least resistance for custom architectures (everything in Phases 3–6). The canonical training loop is worth memorizing:

import torch, torch.nn as nn

model = nn.Sequential(nn.Linear(784, 256), nn.ReLU(),
                      nn.Linear(256, 10))            # logits
opt = torch.optim.AdamW(model.parameters(), lr=1e-3, weight_decay=1e-2)
loss_fn = nn.CrossEntropyLoss()                       # expects raw logits

for epoch in range(epochs):
    model.train()
    for xb, yb in train_loader:
        opt.zero_grad()                # 1. clear old gradients
        logits = model(xb)             # 2. forward pass
        loss = loss_fn(logits, yb)     # 3. compute loss
        loss.backward()                # 4. backprop (autograd)
        opt.step()                     # 5. update parameters
    model.eval()
    with torch.no_grad():
        ...                            # validation, no graph built

TensorFlow / Keras

Worth recognizing: Keras offers a concise high-level API, and TensorFlow has strong production/mobile tooling (TF Lite, TF Serving) — relevant if you ever need on-device inference. You do not need to learn it deeply now; concepts transfer directly from PyTorch.

Checkpoint project

import numpy as np
def softmax(z):
    z = z - z.max(1, keepdims=True)          # numerical stability
    e = np.exp(z); return e / e.sum(1, keepdims=True)

def forward(X, W1, b1, W2, b2):
    Z1 = X @ W1 + b1; A1 = np.maximum(0, Z1)
    Z2 = A1 @ W2 + b2; P = softmax(Z2)
    return Z1, A1, Z2, P

def backward(X, Y, Z1, A1, P, W2):           # Y is one-hot
    n = X.shape[0]
    dZ2 = (P - Y) / n                        # softmax+CE gradient
    dW2 = A1.T @ dZ2;            db2 = dZ2.sum(0)
    dA1 = dZ2 @ W2.T
    dZ1 = dA1 * (Z1 > 0)                     # ReLU derivative
    dW1 = X.T @ dZ1;            db1 = dZ1.sum(0)
    return dW1, db1, dW2, db2

Phase 2 self-check

• Derive backprop for a 2-layer net on paper and pass a numerical gradient check in code.

• Explain why nonlinearities and proper initialization are necessary.

• Contrast SGD, momentum, RMSProp, and Adam, and say when you would use each.

• Explain dropout, weight decay, batch vs. layer norm, and early stopping.

• Write the 5-step PyTorch training loop from memory and explain each line.