Linear regression assumes Gaussian, constant-variance noise; logistic regression handles binary outcomes. Generalized linear models (GLMs) reveal these as two members of one family, unifying regression for continuous, binary, count, and positive responses under a single, elegant template. Understanding the template — a random component, a linear predictor, and a link function — lets you choose the right model for any response type, and clarifies what a neural network’s output layer and loss really are.
The three ingredients
A GLM has three parts:
Random component: the response follows a distribution from the exponential family (Gaussian, Bernoulli, Poisson, Gamma, …).
Linear predictor: \(\eta=\vbeta^\top\vx\), the familiar linear combination.
Link function \(g\): connects the mean to the linear predictor, \(g(\mu)=\eta\), i.e. \(\mu=g^{-1}(\vbeta^\top\vx)\).
The link function is the key generalization. Ordinary regression uses the identity link (\(\mu=\vbeta^\top\vx\)); logistic regression uses the logit link (\(\ln\frac{\mu}{1-\mu}=\vbeta^\top\vx\)); Poisson regression uses the log link (\(\ln\mu=\vbeta^\top\vx\)). One framework, one fitting algorithm (iteratively reweighted least squares), many response types — you just pick the distribution and link that match your data.
The common GLMs
| Response type | Distribution | Link | Model |
|---|---|---|---|
| Continuous | Gaussian | identity | linear regression |
| Binary / proportion | Bernoulli / Binomial | logit | logistic regression |
| Count | Poisson | log | Poisson regression |
| Count (overdispersed) | Negative binomial | log | NB regression |
| Positive, skewed | Gamma | log / inverse | Gamma regression |
Poisson regression is the everyday count model: the log link guarantees a positive mean and turns multiplicative effects additive, so \(e^{\beta_j}\) is a rate ratio — “each unit of \(x_j\) multiplies the expected count by \(e^{\beta_j}\).” Use it for events per unit time/space: website visits, insurance claims, disease incidence. Watch for overdispersion (variance exceeding the mean), where the negative binomial is the fix.
Why the exponential family?
A distribution is in the exponential family if its density can be written \[f(y;\theta)=h(y)\exp\!\big(\theta\, y-A(\theta)\big),\] for a natural parameter \(\theta\) and log-partition function \(A(\theta)\). The Gaussian, Bernoulli, Poisson, Gamma, and many others all fit this mold.
This shared structure is what makes one fitting algorithm work for all GLMs: the likelihood is concave, the canonical link arises naturally, and maximum likelihood reduces to iteratively reweighted least squares — weighted versions of the normal equations from Chapter [ch:multiple].
Two frequent mistakes. (1) Using ordinary linear regression on counts or probabilities — it predicts impossible negative counts or probabilities outside \([0,1]\) and assumes the wrong (constant) variance; reach for the matching GLM instead. (2) Ignoring overdispersion in Poisson models: if the variance far exceeds the mean, standard errors are too small and you will declare spurious significance — switch to negative binomial or use a quasi-Poisson correction.
GLMs are the Rosetta Stone connecting statistics to deep learning’s output layers and losses. A neural network’s final layer plus its loss is exactly a GLM on learned features: a linear output with MSE loss is Gaussian/identity (regression); a sigmoid output with binary cross-entropy is Bernoulli/logit; a softmax output with categorical cross-entropy is the multinomial GLM; and predicting counts with a log-link exponential output and Poisson loss is Poisson regression. Choosing a network’s output activation and loss function is choosing a GLM’s link and distribution. This is why the right loss is not arbitrary — it is the negative log-likelihood of the response’s distribution, exactly as in this chapter.
import numpy as np, statsmodels.api as sm
rng = np.random.default_rng(0)
n = 300
x = rng.normal(size=n)
mu = np.exp(0.5 + 0.8*x) # log link -> positive mean
y = rng.poisson(mu) # count response
X = sm.add_constant(x)
pois = sm.GLM(y, X, family=sm.families.Poisson()).fit()
print(pois.params.round(3)) # ~[0.5, 0.8]
print("rate ratio per unit x:", np.exp(pois.params[1]).round(3)) # exp(beta)Chapter summary
A GLM has a random component (exponential-family distribution), a linear predictor \(\vbeta^\top\vx\), and a link function \(g\).
The link generalizes regression: identity \(\to\) linear, logit \(\to\) logistic, log \(\to\) Poisson.
Counts use Poisson regression (\(e^{\beta_j}\) is a rate ratio); overdispersion calls for the negative binomial.
The exponential-family structure gives one concave likelihood and one fitting algorithm (IRLS).
A neural net’s output activation \(+\) loss is exactly a GLM’s link \(+\) distribution — which is why the loss equals the negative log-likelihood.