ML0005 Discriminative and Generative

Discriminative models learn to draw a boundary between classes: they model the conditional probability directly, mapping features to labels without modeling how the data was generated. Examples include logistic regression, support vector machines, and neural network classifiers. Generative models instead estimate the joint probability (via and ), capturing how the data itself is generated, and then use Bayes’ theorem to derive for classification. Examples include Naive Bayes, hidden Markov models, VAEs, and GANs. Because generative models learn the data distribution, they can also sample new data; discriminative models can only separate existing classes.

(1) Modeling Target: Discriminative models learn directly; generative models learn or .
(2) Decision Mechanism: Discriminative models find a decision boundary; generative models compare class-conditional densities through Bayes’ rule.
(3) Capabilities: Discriminative models only classify; generative models can also generate samples, handle missing features, and score outliers.

Figure 1: Left: a discriminative model cares only about the boundary between classes. Right: a generative model fits a distribution to each class, capturing how the data itself looks.

Mathematical Formulation:

Where:

• is the predicted label and ranges over the class set for input .
• is the posterior, the quantity a discriminative model learns directly.
• is the class-conditional likelihood (how data of class looks) and the class prior; together they define the joint a generative model learns.
• is the evidence normalizer, computable as over all classes.