Discriminative vs Generative Models: When to Use Each in AI

Discriminative Models

Discriminative models learn the conditional distribution P(y\mid x) directly, i.e., the probability of a label y given an input feature vector x . They concentrate on modelling the decision boundary that separates classes and therefore require only enough capacity to distinguish observed samples.

Characteristics

Direct probability learning: Optimises a likelihood or cross‑entropy loss that approximates P(y\mid x).

Efficient training: Gradient‑based optimisation typically converges quickly because the model does not need to model the data distribution P(x).

High predictive accuracy: When abundant labelled data are available, discriminative models often achieve state‑of‑the‑art classification performance.

Flexibility: Can incorporate non‑linear feature transformations, kernels, or deep architectures to handle complex data.

Typical Algorithms

Logistic Regression: Models binary outcomes with a sigmoid function:

\sigma(z)=\frac{1}{1+e^{-z}},\quad P(y=1\mid x)=\sigma(w^{\top}x+b)

Support Vector Machine (SVM): Finds a hyperplane that maximises the margin between classes; training solves a convex quadratic program.

Neural Networks: Stacks multiple linear layers with non‑linear activations; the final softmax layer yields P(y\mid x) for multi‑class problems.

Conditional Random Field (CRF): Extends discriminative modelling to structured outputs (e.g., sequence labeling) by defining potentials conditioned on the entire observation sequence.

Generative Models

Generative models aim to capture the joint distribution P(x, y). By learning how the data are generated, they can derive the conditional distribution via Bayes' theorem:

P(y\mid x)=\frac{P(x, y)}{P(x)} = \frac{P(x\mid y)P(y)}{\sum_{y'}P(x\mid y')P(y')}

This capability enables both classification and data synthesis.

Characteristics

Joint probability modelling: Estimates both P(x\mid y) and the class prior P(y).

Data generation: Can sample new instances x\sim P(x\mid y), useful for augmentation, simulation, or unsupervised learning.

Understanding data structure: By modelling the generative process, these methods expose latent factors and temporal dependencies.

Higher training complexity: Optimisation often involves latent variables, variational bounds, or adversarial objectives, which are computationally more demanding.

Typical Algorithms

Naïve Bayes: Assumes feature independence, computes P(x\mid y)=\prod_i P(x_i\mid y), and classifies by maximizing the posterior.

Hidden Markov Model (HMM): Models sequential data with hidden states z_t and emissions x_t; inference uses the forward‑backward algorithm.

Generative Adversarial Network (GAN): Consists of a generator G(z;θ_g) that maps random noise to synthetic data and a discriminator D(x;θ_d) that distinguishes real from fake samples. Training solves a minimax game:

\min_{θ_g}\max_{θ_d}\;\mathbb{E}_{x\sim P_{data}}[\log D(x)] + \mathbb{E}_{z\sim P_z}[\log(1- D(G(z)))]

Variational Autoencoder (VAE): Learns an encoder q_φ(z\mid x) and decoder p_θ(x\mid z) by maximising the evidence lower bound (ELBO):

\mathcal{L}(θ,φ;x)=\mathbb{E}_{q_φ(z\mid x)}[\log p_θ(x\mid z)] - KL\big(q_φ(z\mid x)\|p(z)\big)

Application Scenarios

Discriminative models: Supervised classification, regression, and anomaly detection (e.g., image or text classification, fraud detection).

Generative models: Data augmentation, realistic image synthesis, speech synthesis, missing‑data imputation, and unsupervised representation learning.