What if the response is a category — spam or not, disease or healthy, click or no click? Fitting a line to \(0/1\) labels gives nonsensical probabilities outside \([0,1]\). Logistic regression fixes this elegantly: it runs a linear model through a squashing function so the output is a valid probability. It is the most important classification method in statistics and — not coincidentally — the exact form of a neuron with a sigmoid activation and the building block of every classification network.

From linear to logistic

Logistic regression models the probability of the positive class by passing a linear predictor through the sigmoid (logistic) function: \[p(\vx)=\Pr(y=1\mid\vx)=\sigma(\vbeta^\top\vx),\qquad \sigma(z)=\frac{1}{1+e^{-z}} .\] The sigmoid maps any real number to \((0,1)\), so predictions are always valid probabilities. Equivalently, the model is linear in the log-odds: \[\logit(p)=\ln\frac{p}{1-p}=\vbeta^\top\vx .\]

The odds \(p/(1-p)\) turn a probability into a ratio (“3 to 1”); the log-odds stretch it to the whole real line so a linear model fits naturally. A coefficient \(\beta_j\) means: a one-unit increase in \(x_j\) adds \(\beta_j\) to the log-odds, i.e. multiplies the odds by \(e^{\beta_j}\). The decision boundary \(\vbeta^\top\vx=0\) (where \(p=0.5\)) is a hyperplane — logistic regression is a linear classifier, even though its probabilities curve.

Fitting: maximum likelihood and cross-entropy

There is no least-squares closed form. Instead we maximize the likelihood of the observed labels, equivalently minimizing the binary cross-entropy (log loss): \[\mathcal{L}(\vbeta)=-\sum_{i=1}^n\Big[y_i\ln p(\vx_i)+(1-y_i)\ln\big(1-p(\vx_i)\big)\Big].\]

Cross-entropy punishes confident wrong predictions brutally: predicting \(p=0.99\) for a true \(0\) incurs near-infinite loss, while a hedged \(p=0.5\) is mild. This is the negative log-likelihood under a Bernoulli model, it is convex (a unique optimum), and there is no closed form — so we optimize numerically by Newton’s method (IRLS) or gradient descent. That last fact is the bridge to neural networks, which minimize this exact loss by gradient descent.

Multiclass: softmax regression

For \(K>2\) classes, softmax (multinomial logistic) regression generalizes the sigmoid: \[\Pr(y=k\mid\vx)=\frac{e^{\vbeta_k^\top\vx}}{\sum_{j=1}^K e^{\vbeta_j^\top\vx}} .\] Each class gets its own coefficient vector; softmax normalizes the scores into a probability distribution that sums to one.

Several traps recur. (1) Perfect separation: if a feature perfectly splits the classes, the MLE coefficients diverge to \(\pm\infty\) — use regularization (a penalty, equivalently a prior) to keep them finite. (2) Threshold \(\ne\) probability: the default \(0.5\) cutoff is rarely optimal under class imbalance or unequal error costs; choose the threshold from a precision–recall or ROC analysis. (3) Accuracy lies on imbalanced data — a 99%-negative dataset is “99% accurate” by always predicting negative; use AUC, F1, or calibrated probabilities instead (Chapter [ch:pitfalls]).

Logistic regression is a single sigmoid neuron, and softmax regression is exactly the final layer of essentially every classification network — from image classifiers to the next-token predictor of a large language model, which is a softmax over the vocabulary. The cross-entropy loss minimized here is the classification loss of deep learning. So a neural classifier is best understood as logistic/softmax regression performed on learned features rather than raw inputs: the hidden layers learn a representation in which the classes are linearly separable, and the output layer is this chapter. Master logistic regression and you understand the head of every deep classifier.

import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.datasets import make_classification
X, y = make_classification(n_samples=400, n_features=5, n_informative=3, random_state=0)
clf = LogisticRegression().fit(X, y)
p = clf.predict_proba(X[:3])[:, 1]
print("P(y=1):", p.round(3))
print("odds multiplier per unit x_j:", np.exp(clf.coef_[0]).round(3))  # exp(beta)

Chapter summary

  • Logistic regression models \(\Pr(y=1\mid\vx)=\sigma(\vbeta^\top\vx)\); it is linear in the log-odds, with \(e^{\beta_j}\) the odds multiplier.

  • The decision boundary is a hyperplane — a linear classifier with curved probabilities.

  • Fit by maximum likelihood \(=\) minimizing cross-entropy (log loss); convex, no closed form, solved by IRLS / gradient descent.

  • Softmax regression extends it to \(K\) classes; watch for perfect separation, threshold choice, and imbalance.

  • It is a sigmoid neuron and the output layer of every deep classifier; cross-entropy is the classification loss of all of ML.