Naive Bayes is Bayes’ theorem turned into a fast, surprisingly effective classifier. Despite an assumption that is almost always false — that features are independent given the class — it powers spam filters, document classifiers, and countless baselines, often beating far more complex models on text. This chapter explains the model, its three flavors, the smoothing that makes it robust, and exactly why “naive” still works.

From Bayes’ theorem to a classifier

To classify an input with features \(\vx=(x_1,\dots,x_d)\) into class \(C\), apply Bayes’ theorem: \[P(C\mid \vx)=\frac{P(\vx\mid C)\,P(C)}{P(\vx)}\ \propto\ P(C)\,P(\vx\mid C).\] The naive assumption is conditional independence of features given the class: \[P(\vx\mid C)=\prod_{j=1}^d P(x_j\mid C),\] so the prediction is \[\hat C=\argmax_{C}\ P(C)\prod_{j=1}^d P(x_j\mid C).\]

The naive assumption replaces one impossibly hard quantity — the joint distribution of all features given the class — with a product of easy one-dimensional ones. Estimating \(P(x_j\mid C)\) for each feature separately needs little data and no high-dimensional density estimation. That is the entire trick: trade a false-but-convenient independence assumption for enormous statistical and computational savings.

Three flavors

Model each \(P(x_j\mid C)\) as a Gaussian; estimate per-class mean and variance. Good for real-valued data.

Model features as word/event counts; \(P(x_j\mid C)\) comes from class word frequencies. The classic for document/topic classification.

Features are present/absent flags; explicitly models non-occurrence too. Good for short texts and presence-of-word features.

Laplace smoothing

If a word never appears in the training spam, \(P(\text{word}\mid\text{spam})=0\), and because the model multiplies probabilities, a single zero annihilates the whole product — one unseen feature vetoes the class. Laplace (add-\(\alpha\)) smoothing fixes this by adding pseudo-counts: \[P(x_j=w\mid C)=\frac{\text{count}(w,C)+\alpha}{\text{count}(C)+\alpha\,V},\] where \(V\) is the vocabulary size. This is exactly a Beta/Dirichlet prior from Chapter [ch:conjugate] — smoothing is Bayesian priors on the word probabilities, ensuring nothing has probability zero just because it was unseen.

Working in log space

Multiplying hundreds of small probabilities causes numerical underflow (the product rounds to zero). Always compute in logs, turning the product into a sum: \[\log P(C\mid\vx)=\log P(C)+\sum_{j=1}^d \log P(x_j\mid C)+\text{const}.\] This is standard practice and the reason library implementations report log-probabilities. Forgetting it silently breaks classification on long documents.

Why “naive” works

The independence assumption is usually false — in text, “New” and “York” are highly dependent. Yet Naive Bayes classifies well because accurate ranking does not require accurate probabilities. Even if the estimated \(P(C\mid\vx)\) is poorly calibrated (often pushed toward 0 or 1 by double-counting correlated features), the argmax — the predicted class — is frequently still correct. Naive Bayes is a low-variance, high-bias model: it cannot overfit much, so with limited data it often beats flexible models, and it trains in a single pass.

Two cautions. (1) Probabilities are unreliable: correlated features make Naive Bayes overconfident, so use it for the decision, not for a trustworthy probability, unless you calibrate. (2) Correlated/redundant features (the same signal entered twice) get double-counted and bias the result — deduplicate features when you can. The class ranking is robust; the numbers are not.

Naive Bayes is the original probabilistic text classifier and remains a standard, hard-to-beat baseline for spam detection, sentiment analysis, and document categorization — fast to train, tiny to store, and effective in high dimensions with little data. In modern NLP pipelines it is the sanity check a fancy Transformer must beat to justify its cost. Conceptually it foreshadows deep generative classifiers: model \(P(\vx\mid C)\) and invert with Bayes. And its smoothing is a clean, intuitive first encounter with the prior-as-pseudo-counts idea that recurs throughout probabilistic ML (e.g. Dirichlet priors in topic models and label smoothing in deep nets).

from sklearn.feature_extraction.text import CountVectorizer
from sklearn.naive_bayes import MultinomialNB
X = ["win money now", "cheap meds online", "let us meet tomorrow",
     "project deadline tomorrow", "free prize claim now", "lunch meeting today"]
y = ["spam", "spam", "ham", "ham", "spam", "ham"]
vec = CountVectorizer(); Xc = vec.fit_transform(X)
clf = MultinomialNB(alpha=1.0).fit(Xc, y)        # alpha = Laplace smoothing (a prior)
test = vec.transform(["free money meeting"])
print(clf.predict(test), clf.predict_proba(test).round(3))

Chapter summary

  • Naive Bayes applies Bayes’ theorem with a conditional-independence assumption: \(P(\vx\mid C)=\prod_j P(x_j\mid C)\).

  • This turns a hard joint density into a product of easy 1-D estimates — cheap, low-variance, great in high dimensions.

  • Flavors: Gaussian (continuous), Multinomial (counts/text), Bernoulli (binary).

  • Laplace smoothing (a Beta/Dirichlet prior) prevents zero probabilities; compute in log space to avoid underflow.

  • It works because correct ranking survives wrong independence assumptions; probabilities are overconfident, so trust the class, not the number. A standard, strong text baseline.