ML0092 Naive Bayes for Continuous Features

The default approach is Gaussian Naive Bayes: for each continuous feature, the model estimates a per-class mean and variance from the training data and plugs the feature value into the Gaussian PDF to get the likelihood. This is fast (one pass to compute mean and variance, O(nd)) and works well when features are approximately normal within each class, as in the Iris dataset or standardized medical lab values. The problem is that the Gaussian assumption is often violated: income is right-skewed, reaction times are log-normal, and sensor readings can be multi-modal. When the true distribution is skewed or has heavy tails, the Gaussian likelihood misestimates the probability mass, pushing the posterior toward 0 or 1 incorrectly. The fixes are discretization (bin the feature and use Categorical or Multinomial NB), kernel density estimation (replace the parametric Gaussian with a non-parametric KDE), or transformation (log or Box-Cox transform to normalize the feature before applying Gaussian NB).

(1) Gaussian NB (Default): estimates and per class per feature, then uses the Gaussian PDF in the likelihood product; O(nd) training, O(d) prediction, but assumes normality.
(2) Discretization: bin the continuous feature into k bins (equal-width or quantile), then treat it as categorical with k values; the 2024 Max-Relevance-Min-Divergence (MRmD) method beats prior discretization schemes on most of 45 benchmark datasets by maximizing discriminant information and generalization simultaneously.
(3) Kernel Density Estimation: replace the Gaussian with a non-parametric KDE ; the R naivebayes package and MATLAB’s ClassificationNaiveBayes both support it, and a 2025 paper used optimized robust KDE with Welsch M-estimation to handle outliers in Bayesian classification.

Figure 1: Three approaches to continuous features in Naive Bayes: Gaussian NB fits a normal curve that misses skewness, discretization bins the feature into categorical probabilities, and kernel density estimation follows the true non-parametric shape.

The practical decision flow is: start with Gaussian NB and check the per-class histograms. If the feature is approximately normal within each class, Gaussian is fine and is the cheapest option. If the feature is skewed but unimodal, apply a log or Box-Cox transform and re-check normality. If the feature is multi-modal or has heavy tails that resist transformation, use discretization (simpler, faster, but loses information) or KDE (more accurate, but slower at prediction because it evaluates the kernel against all training points in that class). Scikit-learn’s GaussianNB also tends to produce overconfident probabilities because the independence assumption compounds with the Gaussian misfit, so CalibratedClassifierCV with isotonic regression is recommended when the predicted probabilities need to be reliable, as in a medical risk score where the probability itself (not just the classification) drives the decision.

Mathematical Formulation:

Where:

• The first equation is Gaussian NB: and are the class-conditional mean and variance of feature , estimated as the sample mean and variance of all training values of feature in class .
• The second equation is kernel density estimation: is the number of training points in class , is the bandwidth, is the kernel function (typically Gaussian), and are the training values of the feature in class .
• For discretization, the continuous range is split into bins, and with Laplace smoothing, reducing the continuous feature to a categorical one.