Information theory, born from Claude Shannon’s 1948 work on communication, turns probability into a measure of information and surprise. Its quantities — entropy, cross-entropy, KL divergence, mutual information — are not exotic add-ons to machine learning; they are its loss functions and objectives. This short chapter makes precise the cross-entropy you have met repeatedly and reveals that training a classifier is, literally, minimizing the information gap between the model and the world.

Entropy: the measure of uncertainty

The entropy of a distribution \(p\) measures its average surprise (in bits, using \(\log_2\), or nats, using \(\ln\)): \[\Hentropy(p)=-\sum_x p(x)\log p(x)=\E_p[-\log p(X)].\]

Define the surprise of an outcome as \(-\log p(x)\): rare events (\(p\) small) are very surprising, certain events (\(p=1\)) are unsurprising (\(-\log 1=0\)). Entropy is the expected surprise — how uncertain you are on average before seeing the outcome. It is maximized by the uniform distribution (maximum ignorance) and zero for a deterministic one (no uncertainty). Shannon showed it is also the minimum average number of bits needed to encode samples from \(p\) — uncertainty and compressibility are the same quantity.

Cross-entropy and KL divergence

The cross-entropy between a true distribution \(p\) and a model \(q\) is the average surprise of \(p\)’s outcomes under \(q\): \[\Hentropy(p,q)=-\sum_x p(x)\log q(x).\] The Kullback–Leibler divergence is the excess: \[\KL(p\,\|\,q)=\sum_x p(x)\log\frac{p(x)}{q(x)}=\Hentropy(p,q)-\Hentropy(p)\ \ge 0,\] with equality iff \(p=q\) (a consequence of Jensen’s inequality, Chapter [ch:limit]).

KL divergence is the “distance” from your model \(q\) to the truth \(p\) — the information you waste by using the wrong distribution. Since \(\Hentropy(p)\) is a fixed property of the data, minimizing cross-entropy over the model \(q\) is exactly minimizing \(\KL(p\,\|\,q)\): pulling the model toward the data distribution. This is why “cross-entropy loss” and “maximum likelihood” and “minimize KL to the data” are three names for the same training objective. KL is not symmetric (\(\KL(p\|q)\neq\KL(q\|p)\)), and which direction you minimize matters: one is mean-seeking, the other mode-seeking.

With a one-hot true label, cross-entropy collapses to \(-\log q(\text{correct class})\) — punishing the model exactly for the probability it failed to assign to the right answer.

Mutual information

The mutual information between \(X\) and \(Y\) is how much knowing one reduces uncertainty about the other: \[\MI(X;Y)=\KL\big(p(x,y)\,\|\,p(x)p(y)\big)=\Hentropy(X)-\Hentropy(X\given Y)\ \ge 0.\]

It is zero exactly when \(X\) and \(Y\) are independent, and unlike correlation it captures nonlinear dependence. It underlies feature selection, representation learning objectives (InfoNCE, the InfoMax principle), and the information-bottleneck view of deep networks.

Information theory supplies the objectives of modern ML almost verbatim. Cross-entropy loss — the default for classification and language modeling — is this chapter’s \(\Hentropy(p,q)\), and minimizing it is maximum likelihood / minimizing \(\KL\) to the data. The ELBO that trains VAEs is an expected log-likelihood minus a \(\KL\) term that regularizes the latent toward its prior. Variational inference minimizes \(\KL\) between an approximate and true posterior. Diffusion models are trained with KL terms between forward and reverse steps. Mutual information drives contrastive self-supervised learning (InfoNCE) and disentanglement. Knowledge distillation matches a student’s distribution to a teacher’s via cross-entropy. When you minimize a loss in deep learning, you are almost always minimizing a divergence between distributions.

import numpy as np
# entropy, cross-entropy, KL (in nats)
p = np.array([0.5, 0.5])
q = np.array([0.9, 0.1])
H  = -(p*np.log(p)).sum()
CE = -(p*np.log(q)).sum()
KL = (p*np.log(p/q)).sum()
print(round(H,3), round(CE,3), round(KL,3))     # CE = H + KL; KL >= 0
print(round(CE - (H + KL), 6))                    # ~0, the identity holds

# classification cross-entropy with a one-hot label collapses to -log q[correct]
q_pred = np.array([0.6, 0.25, 0.1, 0.05]); correct = 0
print(round(-np.log(q_pred[correct]), 3))         # 0.511

Chapter summary

  • Entropy \(\Hentropy(p)=\E[-\log p]\) is average surprise / minimum code length; maximal for uniform, zero for certain.

  • Cross-entropy \(\Hentropy(p,q)\) scores model \(q\) on truth \(p\); KL \(=\Hentropy(p,q)-\Hentropy(p)\ge0\) is the gap, zero iff \(p=q\).

  • Minimizing cross-entropy \(=\) minimizing \(\KL(p\|q)\) \(=\) maximum likelihood — one objective, three names. KL is asymmetric.

  • Mutual information measures (nonlinear) dependence and is zero iff independent.

  • These define classification loss, the VAE ELBO, variational inference, diffusion training, contrastive learning, and distillation.