DL0100 GAN Framework

A Generative Adversarial Network trains two networks against each other. The generator maps a latent noise vector (typically ) to a sample in data space, and the discriminator outputs the probability that its input came from the real dataset rather than from . They share one value function in a two-player minimax game: maximizes the log-likelihood of correctly classifying real and fake batches, while minimizes the same quantity, so the generator’s only learning signal is the gradient that flows backward through the discriminator into the fake samples. The theory is clean: for a fixed the optimal discriminator is the density ratio , and substituting it turns the generator’s objective into the Jensen-Shannon divergence between the data distribution and the model distribution, uniquely minimized when . The practical payoff is that is an implicit model: it never evaluates a likelihood, it just produces a sample in one forward pass, which is why GANs remain attractive whenever sampling latency matters.

(1) Two Networks One Objective: there is a single value function ; the discriminator ascends it and the generator descends it, so no explicit reconstruction or likelihood term is ever written down.
(2) Discriminator As A Density Ratio: the optimal encodes , which is exactly the information the generator needs about where it is over- or under-producing mass.
(3) Implicit Sampling Model: is defined only through the pushforward of by , so sampling is a single forward pass but density evaluation is impossible.
(4) Non-Saturating Generator Loss: the theoretical term has almost no gradient while the generator is bad, so implementations maximize instead.
(5) Alternating Updates, No Loss Curve To Read: training alternates discriminator and generator steps and seeks a saddle point, not a minimum, so a falling loss means nothing and quality is judged with FID or human inspection.

Figure 1: The generator never sees a real sample directly; it only receives gradient that has been routed backward through the discriminator, which is why the quality of bounds what can learn.

A training step draws a minibatch of real samples and a minibatch of latents, updates on both, then updates with held fixed. The original paper allowed discriminator steps per generator step; almost everyone now uses and controls the balance with regularization instead. The three interventions that matter most in practice are spectral normalization or an R1 gradient penalty to keep the discriminator smooth, two-timescale learning rates so the discriminator can stay slightly ahead, and an exponential moving average of the generator weights for evaluation. The classic failure modes are mode collapse, where concentrates on a few outputs that currently fool , and vanishing generator gradient, where a discriminator with near-perfect separation returns almost nothing useful.

Mathematical Formulation:

Where:

• is the shared value function, maximized by and minimized by .
• is a real sample and a latent vector, so is a generated sample and is the estimated probability of being real.
• is the implicit distribution induced by pushing through ; it is never evaluated, only sampled.
• is the optimal discriminator for a fixed , and is the resulting generator criterion; is the Jensen-Shannon divergence, so with equality only at .
• is the non-saturating generator loss actually used in code; it shares the same fixed point as the minimax form but has large gradient when is near 0, which is the required starting condition of training.

Figure 2: Gradient magnitude of the two generator losses with respect to . At the start of training the discriminator wins easily, the saturating form contributes almost nothing, and the non-saturating form is roughly 100x larger at a score of 0.01.