DL0101 GAN Loss Functions

The discriminator optimizes an ordinary binary cross-entropy objective: label real samples 1, generated samples 0, and minimize the negative log-likelihood of that two-class problem. The generator is defined by the same value function played in the opposite direction, so the textbook formulation is the minimax game , but almost nobody implements the generator that way. Minimizing saturates exactly when the generator is bad, which is precisely when it needs signal, so Goodfellow’s original paper already proposed the non-saturating form: maximize , or equivalently minimize . Both variants share the same fixed point, where the optimal discriminator is and the outer objective reduces to the Jensen-Shannon divergence between data and model. In current practice the exact loss shape matters less than the regularizer attached to it, and the common defaults are the non-saturating logistic loss or the hinge loss combined with an R1 gradient penalty on real data.

(1) Discriminator Is Just A Classifier: its loss is standard BCE over a batch of half real and half fake samples, so any logit-based classification loss can be dropped in without changing the game structure.
(2) Saturating Vs Non-Saturating Generator: the two forms have identical optima but very different gradient magnitudes; the non-saturating loss keeps a large gradient when is near 0.
(3) Optimal Discriminator Reveals A Divergence: substituting turns the value function into , which is why a perfect discriminator gives no useful gradient on disjoint supports.
(4) Alternatives Change The Metric: LSGAN uses a squared error, WGAN replaces the classifier with a 1-Lipschitz critic estimating the Earth-Mover distance, and hinge loss caps the penalty once a margin is met.
(5) Regularization Dominates Stability: zero-centered gradient penalties such as R1 are what make training converge locally, and swapping loss families with the same regularizer usually moves FID far less than practitioners expect.

Figure 1: Early in training the discriminator confidently rejects fakes, so sits near 0. At the saturating loss supplies a gradient magnitude of 1.05 while the non-saturating loss supplies 20, about 19x more signal in exactly the regime where the generator is weakest.

Mathematical Formulation:

Where:

• is the value function; the discriminator maximizes it and the generator minimizes it, so is the batch-level binary cross-entropy actually coded up.
• is a real sample and is the estimated probability that its input is real.
• is a latent drawn from a fixed prior, usually ; is the generated sample and the distribution it induces.
• is the literal minimax generator loss and the non-saturating heuristic; they share the same optimum but differ in gradient scale by a factor of .
• is the optimal discriminator for a fixed , obtained by maximizing the integrand pointwise; it equals everywhere when .
• is bounded in , so ranges over and hits its global minimum only at .
• Practical condition: is never reached, since the discriminator takes steps (typically ) per generator step, and the whole derivation assumes optimization in function space rather than over network parameters.

The main alternatives keep the same alternating structure and change only the per-sample penalty. LSGAN replaces the log with a squared error, which pushes fake samples toward the decision boundary and stops the loss from flattening, at the cost of also penalizing samples that are already comfortably on the correct side. WGAN drops the sigmoid entirely: the critic maximizes under a 1-Lipschitz constraint, enforced in WGAN-GP by a two-sided penalty on the gradient norm at interpolated points, and the generator simply maximizes . Hinge loss, the default in BigGAN and many diffusion-era discriminators, uses on reals and on fakes, so a correctly classified sample beyond the margin contributes exactly zero gradient and the discriminator stops overfitting easy examples. Independently of the family chosen, the R1 penalty is what turns the local dynamics from oscillatory into convergent.

Figure 2: Loss contributed by one fake sample as a function of its logit . BCE uses and decays smoothly but never quite to zero, hinge is exactly flat once the sample clears the margin, and the Wasserstein critic stays linear so it never saturates and must be constrained by a Lipschitz penalty.