DL0102 Diffusion vs GAN vs VAE

A diffusion model defines a fixed forward process that gradually corrupts data into Gaussian noise over steps, then learns the reverse process that removes a little noise at a time. The forward chain has no parameters and admits a closed form, so training reduces to a plain regression: sample a clean image , sample a timestep and noise , build the noisy state , and ask the network to predict . This denoising objective is a weighted variational bound on the log-likelihood and is also equivalent to score matching, which is why the same network can be plugged into an SDE or ODE sampler. The practical consequence is that diffusion buys stable training and excellent mode coverage at the price of many network evaluations per sample, whereas a GAN generates in one forward pass but fights an unstable min-max game, a VAE trains stably but produces blurry samples under a Gaussian decoder, and a normalizing flow gives exact likelihoods but pays for invertibility with architectural constraints and parameter count.

(1) Fixed Corruption, Learned Reversal: only the reverse direction has parameters, so there is no adversary and no discriminator to balance, which removes the main source of GAN training instability.
(2) Closed-Form Training Target: because is Gaussian, any timestep can be sampled directly without simulating the chain, making each training step in .
(3) Score Interpretation: the noise predictor is a rescaled estimate of , which links diffusion to Langevin dynamics and to probability-flow ODE samplers such as DDIM.
(4) Sampling Is The Bottleneck: quality scales with the number of function evaluations, so a 1000-step DDPM sampler costs three orders of magnitude more compute per image than a GAN generator.
(5) Coverage Versus Speed: the likelihood-flavored objective penalizes dropped modes, which is exactly the failure GANs are known for, so the two families sit on opposite corners of the quality/diversity/speed trade-off.

Figure 1: The forward process is fixed and parameter-free, and the dashed arc is the closed form that lets training jump straight to any ; every arrow in the lower reverse path costs one network evaluation at sampling time.

The four families differ mostly in how they pay for tractability. A VAE keeps an explicit encoder and decoder and optimizes a lower bound, so the mismatch between the true posterior and the Gaussian approximation shows up as blur and posterior collapse. A normalizing flow refuses any approximation and computes exact likelihoods, but every layer must be invertible with a cheap Jacobian determinant, which caps expressiveness per parameter. A GAN drops likelihood entirely and optimizes a discriminator-defined divergence, which produces sharp samples in one step but offers no signal that the whole data distribution is covered. Diffusion sidesteps all three constraints by turning generation into a long sequence of easy denoising problems, and then spends compute at inference to pay for it.

Mathematical Formulation:

Where:

• is a clean data sample and the noised state at step , with approximately standard Gaussian.
• is the injected noise and the learned noise predictor with parameters .
• is sampled uniformly during training, and indexes the product defining the cumulative signal retention.
• is the noise schedule, typically small and increasing, and is the cumulative signal factor giving signal-to-noise ratio .
• is the implied score function, the quantity an SDE or probability-flow ODE sampler integrates.
• Required initial condition: sampling starts from , which is only valid if the schedule drives to (near) zero.

Figure 2: The generative learning trilemma: each family wins two corners and pays on the third, and most modern work on diffusion (distillation, consistency training, rectified flow) is an attempt to drag it toward the fast-sampling corner without losing coverage.