ML0090 Naive Bayes Classifier

Naive Bayes is a probabilistic classifier that applies Bayes’ theorem with a “naive” assumption: given the class label, all features are conditionally independent. This lets the classifier multiply per-feature likelihoods instead of estimating the full joint distribution, which would require exponential data. Despite the assumption being almost always violated in practice, Naive Bayes trains in O(nd) time, predicts in O(d) time, and performs surprisingly well on text classification, spam filtering, and support ticket routing. A production case study processes 85,000 product reviews per day with Multinomial Naive Bayes, achieving AUC above 0.99 for spam detection while retraining in under 5 minutes on a single-core VM, and a TF-IDF plus Multinomial NB pipeline routes banking support tickets in under 1 millisecond per ticket on the Banking77 dataset.

(1) Bayes’ Theorem Applied: the classifier computes the posterior for each class and picks the argmax; the naive assumption factors the likelihood into a product of one-dimensional terms .
(2) Why the Assumption Works Anyway: even when the independence assumption is wrong, the argmax decision is often correct because correlated features push the posterior in the same direction; the calibration is off but the ranking is not, which is why scikit-learn recommends CalibratedClassifierCV with isotonic regression for reliable probabilities.
(3) Production Strengths: trains in milliseconds, predicts in microseconds, needs no GPU, produces interpretable per-feature contributions, and a 2024 Generalized Naive Bayes paper showed that relaxing the independence assumption via optimal structure learning improves accuracy while keeping the computational profile.

Figure 1: The naive independence assumption factorizes the intractable joint likelihood P(x1, x2, x3 | y) into a product of one-dimensional per-feature likelihoods, replacing an exponential-size table with d small tables.

The independence assumption is the single most important thing to understand about Naive Bayes, because it is both its source of speed and its source of error. When two features are highly correlated (e.g., “free” and “gift” both appearing in spam), the classifier double-counts their evidence, pushing the posterior toward 0 or 1 more aggressively than the true joint distribution would. This is why GaussianNB in scikit-learn tends to produce overconfident probabilities and why calibration is recommended. The assumption also means Naive Bayes cannot capture feature interactions; a 2024 paper on Generalized Naive Bayes (GNB) addresses this by learning an optimal dependency structure among features while preserving the efficient factorization, proving the approximation is at least as good as classical Naive Bayes. For text specifically, the classic remedies are TF-IDF weighting, document-length normalization, and complement Naive Bayes, introduced by Rennie et al. to correct the multinomial model’s mismatch with real text. In practice, the speed-accuracy trade-off favors Naive Bayes for high-volume, low-latency text tasks where a transformer would be overkill.

Mathematical Formulation:

Where:

• is the predicted class; is the class prior, and the product of per-feature likelihoods replaces the intractable joint under the naive conditional independence assumption.
• The second equation is the smoothed estimator for discrete (count) features: is the count of feature in class , is the total count for class , is the vocabulary size, and is the Laplace smoothing parameter (typically 1) that prevents zero probabilities for unseen feature-class combinations. Continuous features use a density instead, such as a per-class Gaussian.
• The argmax is computed in log space to avoid underflow: , which is a sum of precomputed log-likelihoods and runs in O(d) per prediction.