Tag: Loss

DL0186 Energy-Based Models and Contrastive Divergence

What is the Energy-Based Model (EBM) formulation, and how does Contrastive Divergence approximate the intractable partition function gradient without sampling full Markov chains?

Answer

An energy-based model refuses to parameterize a normalized density directly. It learns only a scalar energy function $E_{\theta}(x)$ , low for plausible configurations and high for implausible ones, and recovers a density by exponentiating and normalizing. The price of that freedom is the partition function $Z(\theta)$ , an integral over the whole input space that is intractable for anything beyond a toy problem. Differentiating the log-likelihood splits into a positive phase that lowers energy at observed data and a negative phase that raises energy wherever the model currently places mass, and the negative phase is an expectation under $p_{\theta}$ that normally needs MCMC samples at equilibrium. Contrastive divergence keeps the same two-phase update but initializes the chain at the training data rather than at noise, then runs only $k$ transition steps, in practice $k = 1$ . The estimate is biased because the $k$ -step distribution is not the equilibrium one, yet the bias shrinks near the optimum where the data distribution is already close to a fixed point of the sampler, and the variance is far lower than that of a long chain.

(1) Energy, Not Probability: the network outputs one unconstrained scalar per input, so any architecture works and no normalization layer or invertibility constraint is required.
(2) The Partition Function Is The Whole Problem: $Z(\theta)$ is a sum or integral over every possible configuration, so exact likelihood and exact sampling are both out of reach.
(3) Positive And Negative Phase: the exact gradient is a difference of two expectations of $\nabla_{\theta}E$ , one under the data and one under the model, which is why training is often described as pushing energy down at real points and up at fantasy points.
(4) Chain Started At The Data: CD replaces the equilibrium expectation with samples from a chain seeded at $x^{+}$ , which already sits in a high-probability region, so a very short run produces a usable contrast.
(5) Truncation Trades Bias For Cost: per-update cost drops from a full mixing time to $k$ transitions, at the cost of a systematically biased gradient that ignores model mass far from the data.
(6) CD Is An Update Rule, Not A True Gradient: the CD-1 update is not the exact gradient of any objective function, so convergence guarantees are weak and divergence is possible in principle.

Flow diagram with a top lane showing a data batch feeding the positive phase that lowers energy at the data point, and a bottom lane where the same data point initializes a Markov chain of k short steps producing the negative sample, with both phases feeding a single parameter update box

Figure 1: The two-phase update. Exact maximum likelihood would draw the negative sample from the model at equilibrium, costing a full chain run per parameter update. CD keeps the identical update form but seeds the chain at the data and stops after $k$ steps, which turns an expensive unbiased gradient into a cheap biased one.

Mathematical Formulation:
$p_{\theta}(x) = \frac{e^{-E_{\theta}(x)}}{Z(\theta)}$
$Z(\theta) = \int e^{-E_{\theta}(x)}\,dx$
$\log p_{\theta}(x) = -E_{\theta}(x) - \log Z(\theta)$
$\nabla_{\theta}\log Z(\theta) = -\mathbb{E}_{p_{\theta}}[\nabla_{\theta}E_{\theta}]$
$\nabla_{\theta}\mathcal{L} = \mathbb{E}_{p_{\mathrm{data}}}[\nabla_{\theta}E] - \mathbb{E}_{p_{\theta}}[\nabla_{\theta}E]$

Where:

$p_{\theta}(x)$ is the model density and $E_{\theta}(x) \in \mathbb{R}$ the learned energy, with lower energy meaning higher probability.
$Z(\theta)$ is the partition function, an integral over the full input space (a sum for discrete $x$ ) that depends on $\theta$ and therefore cannot be ignored during training.
$x$ is a configuration and $\theta$ the parameters of the energy network.
$\mathcal{L} = -\mathbb{E}_{p_{\mathrm{data}}}[\log p_{\theta}]$ is the average negative log-likelihood, so the first expectation is the positive phase and the second is the negative phase.
The identity $\nabla_{\theta}\log Z = -\mathbb{E}_{p_{\theta}}[\nabla_{\theta}E]$ is what converts an intractable normalizer into an expectation, which is exactly why sampling becomes the bottleneck.

The Contrastive Divergence Approximation:
$p_k = p_0 T_{\theta}^{k}$
$\nabla_{\theta}\mathcal{L}_{\mathrm{CD}} = \mathbb{E}_{p_0}[\nabla_{\theta}E] - \mathbb{E}_{p_k}[\nabla_{\theta}E]$
$\mathcal{L}_{\mathrm{CD}} \approx D_{KL}(p_0 \| p_{\theta}) - D_{KL}(p_k \| p_{\theta})$

Where:

$p_0 = p_{\mathrm{data}}$ is the empirical distribution used as the chain initialization, and $p_k$ is the distribution after $k$ applications of the transition operator.
$T_{\theta}$ is any transition operator leaving $p_{\theta}$ invariant, typically block Gibbs for an RBM or Langevin dynamics for a continuous EBM.
$k$ is the truncation depth, with $k = 1$ the standard choice and larger $k$ reducing bias at linear cost.
As $k \to \infty$ we recover $p_k = p_{\theta}$ and the update becomes the exact likelihood gradient, so CD is a controllable interpolation between a free estimate and an unbiased one.
The two-KL form is the original motivation rather than an exact objective; a small ignored term involving $\partial p_k / \partial \theta$ is what makes CD not the gradient of any function.

The restricted Boltzmann machine is the case where every piece is closed-form, which is why CD was invented there. Because there are no visible-visible or hidden-hidden couplings, the conditionals factorize completely and one Gibbs sweep is a single pass of matrix multiplies plus sigmoids, following the chain v → h → v’ → h’. CD-1 then reduces to the difference between a data correlation and a reconstruction correlation, computable with two forward passes and no backpropagation through a sampler.

Binary RBM Case:
$E_{\theta}(v,h) = -v^{\top}Wh - b^{\top}v - c^{\top}h$
$p(h_j = 1 \mid v) = \sigma(c_j + W_{:j}^{\top}v)$
$\Delta W = \eta\,(v_0 h_0^{\top} - v_1 h_1^{\top})$

Here $v_0$ is the data vector, $h_0 \sim p(h \mid v_0)$ , and $(v_1, h_1)$ come from one further Gibbs sweep. The positive term raises the weight between co-active data units and the negative term subtracts the same statistic measured on the model’s one-step reconstruction, so learning stops precisely when the reconstruction statistics match the data statistics.

One-dimensional energy landscape curve with a deep data basin on the left and a spurious model basin on the right separated by a barrier, showing the data point where the positive phase lowers energy, the k-step chain sample nearby where the negative phase raises energy, and a distant full-chain sample in the spurious basin that a short chain never reaches

Figure 2: The structural blind spot. A $k$ -step chain started at $x^{+}$ stays inside the data basin, so the negative phase only ever penalizes points the model already ranks near the data. A spurious mode behind an energy barrier keeps its low energy indefinitely because CD never samples there, and this is why CD-trained models can assign high likelihood to garbage.

The practical consequences follow directly from that picture. Slow mixing is not a nuisance but the defining constraint, since the time to cross a barrier grows exponentially in its height, so raising $k$ from 1 to 20 buys much less than the cost suggests. Persistent CD attacks the same problem differently by keeping a set of chains alive across parameter updates instead of restarting them at data, which lets the fantasy particles drift into spurious regions given enough updates, at the cost of instability when the learning rate is large relative to the mixing rate. Modern continuous EBMs use short-run Langevin dynamics with a replay buffer plus gradient clipping and energy regularization, and increasingly avoid MCMC entirely through score matching or its denoising variant, which never touches $Z(\theta)$ because the score $\nabla_x \log p_{\theta}(x) = -\nabla_x E_{\theta}(x)$ is independent of the normalizer.

Property	CD-k	Persistent CD	Score matching
Negative sample source	k transitions from the current data batch	Persistent fantasy particles carried across updates	None; no sampling during training
Cost per update	k sweeps, usually one	One sweep, plus buffer memory	One or two extra backward passes for derivatives
Main bias or error	Truncation bias; distant spurious modes never penalized	Chains lag a moving target, so samples are stale	Consistent but blind to relative mass between separated modes
Stability	Very stable, low variance, forgiving of large learning rates	Needs a small learning rate so chains can keep up	Stable, but plain score matching struggles in high dimension without noise
Best fit	RBMs and discrete models with exact block Gibbs	Longer training runs where density quality matters more than speed	Continuous high-dimensional data, the route taken by diffusion models

August 20, 2026

DL0185 CycleGAN Cycle Consistency Loss

How does CycleGAN use Cycle Consistency Loss to train unpaired image-to-image translation, and what are its limitations regarding mode collapse and semantic preservation?

Answer

CycleGAN trains two generators at once, $G: X \to Y$ and $F: Y \to X$ , each with its own PatchGAN discriminator, using only samples from the two marginals $p_X$ and $p_Y$ and never a matched pair. The adversarial terms alone are hopelessly under-constrained, because any mapping whose pushforward equals $p_Y$ is optimal, including one that ignores the input entirely. The cycle consistency loss adds an $\ell_1$ reconstruction penalty in both directions, $F(G(x)) \approx x$ and $G(F(y)) \approx y$ , which forces the two generators to be approximate inverses and therefore rules out constant outputs and arbitrary re-pairing. The critical limitation is that this constraint is invariant to any label-permuting bijection: it enforces invertibility, not meaning, so a translation that swaps content while remaining perfectly reversible pays zero cycle loss. In practice this shows up as residual per-mode collapse (one stripe pattern for every horse), self-adversarial steganography (imperceptible high-frequency codes that let $F$ cheat), and semantic hallucination such as painting zebra texture onto a rider or inventing a lesion in a medical scan.

(1) Two Coupled Mappings, Four Networks: $G$ , $F$ , $D_Y$ , and $D_X$ train jointly, with the adversarial terms fixing the target marginal and nothing else.
(2) Cycle Loss As A Soft Bijection: a bidirectional $\ell_1$ reconstruction penalty weighted by $\lambda = 10$ ties the two generators into an approximate inverse pair.
(3) What It Genuinely Prevents: total collapse is impossible, since a generator that emits one fixed image destroys all input information and $F$ cannot reconstruct $x$ from it.
(4) What It Does Not Constrain: composing $G$ with any measure-preserving permutation of the target domain leaves both losses unchanged, so semantics is unconstrained and only the convolutional inductive bias keeps translations sane.
(5) Self-Adversarial Steganography: the pair can hide a low-amplitude code in the output so reconstruction is near-perfect while the visible translation is wrong, making cycle loss a weak proxy for content preservation.
(6) Deterministic One-To-One Bias: cycle consistency actively penalizes information loss, so many-to-one tasks and geometric or shape changes fail while texture and color remapping succeeds.

Diagram of the CycleGAN objective with a top row showing a real horse photo passing through generator G to a fake zebra and then through generator F back to a reconstruction, a bottom row showing the reverse cycle from a real zebra through F and G, dashed routed paths marking the two L1 cycle losses, and two discriminator boxes in the middle band receiving the fake samples

Figure 1: The full objective in one view. The adversarial terms only ask whether $G(x)$ looks like some member of domain $Y$ , and the cycle terms only ask whether the round trip returns to the start, so no term in the loss ever compares a translated horse against the true zebra version of that same horse.

The published recipe matters because most of the stability comes from implementation choices rather than from the loss on paper. The generators are ResNet encoders with 9 residual blocks at $256 \times 256$ resolution, the discriminators are $70 \times 70$ PatchGAN critics that score local patches instead of the whole image, and the adversarial term uses the least-squares (LSGAN) form rather than the log-likelihood form to avoid vanishing gradients. Discriminators are updated against a buffer of 50 previously generated images, which damps oscillation, and training uses Adam at $2 \times 10^{-4}$ with batch size 1 and a linear learning-rate decay after the first 100 epochs. For photo and painting transfer an extra identity loss weighted at $0.5 \lambda$ penalizes $\|G(y) - y\|_1$ , which is a purely empirical patch that stops the generators from inverting the color palette.

Mathematical Formulation:
$\mathcal{L}_{\mathrm{GAN}}(G, D_Y) = \mathbb{E}_y[(D_Y(y) - 1)^2]$
$+ \mathbb{E}_x[D_Y(G(x))^2]$
$\mathcal{L}_{\mathrm{cyc}} = \mathbb{E}_x \| F(G(x)) - x \|_1$
$+ \mathbb{E}_y \| G(F(y)) - y \|_1$
$\mathcal{L} = \mathcal{L}_{\mathrm{GAN}}(G, D_Y) + \mathcal{L}_{\mathrm{GAN}}(F, D_X)$
$+ \lambda \mathcal{L}_{\mathrm{cyc}}$

Where:

$\mathcal{L}_{\mathrm{GAN}}(G, D_Y)$ is the least-squares adversarial term that $D_Y$ minimizes; the generator instead minimizes $\mathbb{E}_x[(D_Y(G(x)) - 1)^2]$ , which is the non-saturating counterpart.
$x \sim p_X$ and $y \sim p_Y$ are independent draws from the two marginals; there is no joint distribution over pairs anywhere in the objective.
$G$ and $F$ are the two generators, $D_X$ and $D_Y$ the two patch discriminators, all four trained simultaneously.
$\mathcal{L}_{\mathrm{cyc}}$ sums the forward and backward reconstruction errors in $\ell_1$ , chosen over $\ell_2$ because it blurs less.
$\lambda$ is the cycle weight, set to 10 in the paper; raising it sharpens reconstruction and suppresses collapse, but it also makes the generator conservative and can push it toward hidden encodings.

Why The Constraint Is Semantically Blind:
$G' = T \circ G$
$F' = F \circ T^{-1}$
$T_{\#} p_Y = p_Y$
$\mathcal{L}(G', F') = \mathcal{L}(G, F)$

Take any invertible map $T$ on the target domain whose pushforward leaves $p_Y$ unchanged, for example a permutation that sends every image of a single zebra to an image of two zebras and vice versa. Then $G' = T \circ G$ produces samples with exactly the same distribution, so both discriminators are equally fooled, and $F' = F \circ T^{-1}$ still reconstructs the input exactly, so the cycle loss is unchanged. The two solutions are therefore indistinguishable global optima, and the one you actually get is decided by the generator’s limited receptive field, its residual skip structure, and the fact that near-identity mappings are easy to learn. That is a statement about architecture, not about the loss, which is why CycleGAN is reliable for texture and color transfer and unreliable the moment a task requires moving or removing structure.

Two side-by-side panels each showing three boxes of horse-photo content classes on the left and three boxes of zebra-photo content classes on the right; the left panel connects them with parallel content-preserving arrows while the right panel connects them with crossing permuted arrows, and both panels are annotated with identical zero cycle loss and matched marginals

Figure 2: Two mappings between the same two sets. Both are exact bijections and both match the target marginal, so adversarial loss and cycle loss are numerically identical, yet only the left one preserves content. Nothing in the CycleGAN objective distinguishes them, which is the formal root of label flipping and hallucination.

Property	CycleGAN (cycle loss)	CUT (patchwise contrastive)	CycleGAN-Turbo (one-step diffusion)
Content constraint	Round-trip $\ell_1$ reconstruction in both directions	PatchNCE: a patch must match its own location across the translation	Cycle plus perceptual identity on top of a pretrained text-to-image prior
Networks trained	Two generators plus two discriminators	One generator, one discriminator, one small projection head	LoRA adapters on a frozen one-step diffusion backbone
Training cost	Highest of the three: four forward passes per step	Roughly half the memory, single-direction training	Low trainable parameter count, but a large frozen backbone in memory
Output diversity	Deterministic, one output per input by construction	Deterministic, but no invertibility pressure to preserve everything	Deterministic at one step, though the prior supplies plausible detail
Characteristic failure	Hidden encodings, label flipping, refusal to change shape	Weaker global structure retention when the patch head is mistuned	Inherits the prior’s biases and hallucinates plausible but absent detail

August 20, 2026

DL0181 Label Smoothing and Model Calibration

What is Label Smoothing, and how does it affect model calibration, overconfidence, and the trade-off between accuracy and log-likelihood?

Answer

Label smoothing replaces the one-hot target with a mixture of the one-hot vector and the uniform distribution, so the true class receives $1-\epsilon+\epsilon/K$ and every other class receives $\epsilon/K$ . Written this way it is exactly standard cross-entropy plus an $\epsilon$ -weighted KL term pulling the prediction toward uniform, which gives the loss a finite minimizer instead of one that is reached only as the true-class probability approaches 1. That single change removes the pressure that makes logit gaps grow without bound after the argmax is already correct, so the model stops becoming more confident with more training and the classic overconfidence of modern networks largely disappears. The consequences are asymmetric: Expected Calibration Error (ECE) usually drops sharply and top-1 accuracy is flat or slightly better, but the hard-label negative log-likelihood gets worse, because a converged smoothed model deliberately withholds probability mass from the correct class. The Transformer paper states this trade-off explicitly, using $\epsilon=0.1$ and noting that it hurts perplexity while improving accuracy and BLEU.

(1) Target Mixing: the label becomes $(1-\epsilon)y+\epsilon/K$ , which is equivalent to cross-entropy plus $\epsilon\,\mathrm{KL}(u\,\|\,p)$ up to an additive constant.
(2) Bounded Logit Gap: the optimum sits at a finite logit difference, roughly 9.1 nats for $\epsilon=0.1$ and $K=1000$ , so logit norms stop inflating.
(3) Confidence Ceiling: a well-fit smoothed model cannot report top-class confidence above $1-\epsilon+\epsilon/K$ , which is 0.900 at $\epsilon=0.1$ .
(4) Calibration Improves, Then Overshoots: ECE is U-shaped in $\epsilon$ , moving the model from overconfident through calibrated to systematically underconfident.
(5) Log-Likelihood Penalty: the confidence ceiling implies a hard-label NLL floor of $-\log(1-\epsilon+\epsilon/K)$ , about 0.105 nats per example at $\epsilon=0.1$ .
(6) Information Erasure: smoothing equalizes the wrong-class logits, which tightens penultimate-layer clusters and damages knowledge distillation and feature transfer.

Reliability diagram plotting bin accuracy against mean predicted confidence for three training settings: hard labels sit below the perfect-calibration diagonal indicating overconfidence, label smoothing with epsilon 0.1 tracks the diagonal closely and stops at confidence 0.900, and label smoothing with epsilon 0.3 sits above the diagonal indicating underconfidence and stops at confidence 0.700

Figure 1: Reliability curves for the same architecture under three targets. Hard labels fall below the diagonal (confidence exceeds accuracy), $\epsilon=0.1$ tracks the diagonal, and $\epsilon=0.3$ lands above it. Note the truncated horizontal extent: each smoothed curve stops at its confidence ceiling $1-\epsilon+\epsilon/K$ , 0.900 and 0.700 for $K=1000$ . Curves are illustrative of published trends rather than a single benchmark run.

The mechanism is easiest to read off the gradient with respect to the true-class logit, which is $p_y-1$ under hard labels and $p_y-(1-\epsilon+\epsilon/K)$ under smoothing. The hard-label gradient is strictly negative for every finite logit, so optimization keeps pushing the correct logit away from the others long after the prediction is right, and confidence keeps drifting upward while accuracy has already plateaued. The smoothed gradient changes sign once the model reaches the target probability, which pins confidence and caps the logit gap. The wrong-class gradients change too: each incorrect logit is pulled toward the same small target $\epsilon/K$ , so the model is discouraged from expressing that “husky” is much closer to “wolf” than to “airliner”. That equalization is exactly the property Müller and colleagues identified as the reason a label-smoothed teacher makes a worse distillation teacher than a hard-label teacher of identical accuracy.

Mathematical Formulation:
$y^{\mathrm{LS}}_k = (1-\epsilon)\,y_k + \epsilon/K$
$\mathcal{L}_{\mathrm{LS}} = -\sum_{k=1}^{K} y^{\mathrm{LS}}_k \log p_k$
$\mathcal{L}_{\mathrm{LS}} = (1-\epsilon)\mathcal{L}_{\mathrm{CE}} + \epsilon\,\mathrm{KL}(u\,\|\,p) + c$
$p^{\star}_{y} = 1 - \epsilon + \epsilon/K$
$z_y - z_j = \log\left(\frac{K(1-\epsilon)+\epsilon}{\epsilon}\right)$
$\mathrm{ECE} = \sum_{m=1}^{M} \frac{|B_m|}{N}\left|\mathrm{acc}(B_m) - \mathrm{conf}(B_m)\right|$

Where:

$y \in \{0,1\}^{K}$ is the one-hot label and $y^{\mathrm{LS}}$ the smoothed target over $K$ classes.
$\epsilon \in [0,1)$ is the smoothing strength, $\epsilon=0$ recovering ordinary cross-entropy; $0.1$ is the near-universal default.
$p_k = \mathrm{softmax}(z)_k$ is the predicted probability, $z$ the logits, $y$ the index of the correct class and $j$ any incorrect class.
$u_k = 1/K$ is the uniform distribution, and $c = \epsilon H(u)$ is a constant independent of the parameters, which is why the objective is cross-entropy plus a uniform-KL penalty.
$p^{\star}_{y}$ is the per-example minimizer, giving the finite logit gap in the fifth line; the gap grows only logarithmically as $\epsilon$ shrinks.
$B_m$ is the set of the $N$ validation examples whose top confidence falls in bin $m$ of $M$ (typically $M=15$ ), and ECE is the confidence-weighted gap plotted in Figure 1.

Log-Likelihood Floor At $\epsilon=0.1$ , $K=1000$ :
$p^{\star}_{y} = 0.9 + 0.0001 = 0.9001$
$-\log(0.9001) = 0.1052$
$\exp(0.1052) = 1.111$

The third line is the practical reading for sequence models: a converged smoothed model pays roughly 11% higher perplexity than its own hard-label counterpart would at the same accuracy, purely from the withheld mass, which is why translation systems that smooth at training time are usually evaluated with BLEU or COMET rather than perplexity. Note also that this floor is a property of the optimum, not a hard constraint on the network: a single global temperature fitted on held-out data can sharpen the smoothed logits back and recover most of the lost likelihood. That observation is the reason smoothing is not the right tool if calibrated probabilities are the goal, since post-hoc temperature scaling reaches comparable ECE at zero training cost, one scalar parameter, and no distortion of the wrong-class ranking.

Three side-by-side line charts against smoothing strength epsilon from 0 to 0.4: hard-label negative log-likelihood dips slightly then rises while an analytic floor curve rises monotonically, expected calibration error forms a U shape with a minimum near epsilon 0.1, and top-1 accuracy stays on a flat plateau before declining past epsilon 0.2

Figure 2: The three metrics disagree about the best $\epsilon$ . ECE is U-shaped with a minimum near $0.05$ to $0.1$ , hard-label NLL dips only briefly before the analytic floor $-\log(1-\epsilon+\epsilon/K)$ dominates, and accuracy is flat across a broad plateau. Choosing $\epsilon$ therefore means choosing which metric you are optimizing. Values are illustrative of published sweeps.

Property	Label smoothing	Temperature scaling	Confidence penalty / focal loss
When applied	Training time, changes the targets	Post-hoc, one scalar fitted on a held-out split	Training time, changes the loss on the prediction side
Effect on accuracy	Flat to slightly better; degrades past roughly 0.2	Exactly zero, the argmax is invariant to a positive temperature	Task dependent; focal loss helps mainly under heavy class imbalance
Effect on hard-label NLL	Worse at convergence, floored at $-\log(1-\epsilon+\epsilon/K)$	Directly minimized by the fitting objective, so it improves	Usually worse, for the same mass-withholding reason
Logit ranking preserved	No, wrong-class logits are equalized	Yes, it is a monotone rescaling of all logits	Partially, the entropy term flattens the tail
Main failure mode	Underconfidence at large $\epsilon$ ; weaker distillation teacher and weaker transfer features	One global scalar cannot fix per-class or per-slice miscalibration, and it needs a clean validation split	Extra hyperparameter with little calibration gain over a fitted temperature

August 20, 2026

DL0101 GAN Loss Functions

What loss functions do the GAN generator and discriminator optimize?

Answer

The discriminator optimizes an ordinary binary cross-entropy objective: label real samples 1, generated samples 0, and minimize the negative log-likelihood of that two-class problem. The generator is defined by the same value function played in the opposite direction, so the textbook formulation is the minimax game $\min_{G} \max_{D} V(D,G)$ , but almost nobody implements the generator that way. Minimizing $\log(1 - D(G(z)))$ saturates exactly when the generator is bad, which is precisely when it needs signal, so Goodfellow’s original paper already proposed the non-saturating form: maximize $\log D(G(z))$ , or equivalently minimize $-\log D(G(z))$ . Both variants share the same fixed point, where the optimal discriminator is $D^{*}(x) = p_{r}(x)/(p_{r}(x)+p_{g}(x))$ and the outer objective reduces to the Jensen-Shannon divergence between data and model. In current practice the exact loss shape matters less than the regularizer attached to it, and the common defaults are the non-saturating logistic loss or the hinge loss combined with an R1 gradient penalty on real data.

(1) Discriminator Is Just A Classifier: its loss is standard BCE over a batch of half real and half fake samples, so any logit-based classification loss can be dropped in without changing the game structure.
(2) Saturating Vs Non-Saturating Generator: the two forms have identical optima but very different gradient magnitudes; the non-saturating loss keeps a large gradient when $D(G(z))$ is near 0.
(3) Optimal Discriminator Reveals A Divergence: substituting $D^{*}$ turns the value function into $2\,\mathrm{JSD}(p_{r} \,\|\, p_{g}) - \log 4$ , which is why a perfect discriminator gives no useful gradient on disjoint supports.
(4) Alternatives Change The Metric: LSGAN uses a squared error, WGAN replaces the classifier with a 1-Lipschitz critic estimating the Earth-Mover distance, and hinge loss caps the penalty once a margin is met.
(5) Regularization Dominates Stability: zero-centered gradient penalties such as R1 are what make training converge locally, and swapping loss families with the same regularizer usually moves FID far less than practitioners expect.

Two panel chart: left panel plots the saturating generator loss log of one minus D and the non-saturating loss minus log D against the discriminator score on a fake sample, right panel plots the magnitude of their gradients on a logarithmic axis showing the non-saturating gradient is far larger when the score is near zero

Figure 1: Early in training the discriminator confidently rejects fakes, so $D(G(z))$ sits near 0. At $D(G(z)) = 0.05$ the saturating loss supplies a gradient magnitude of 1.05 while the non-saturating loss supplies 20, about 19x more signal in exactly the regime where the generator is weakest.

Mathematical Formulation:
$\min_{G} \max_{D} V(D, G)$
$V(D,G) = \mathbb{E}_{x \sim p_{r}}[\log D(x)]$
$+ \mathbb{E}_{z \sim p_{z}}[\log (1 - D(G(z)))]$
$\mathcal{L}_{G}^{\mathrm{sat}} = \mathbb{E}_{z}[\log (1 - D(G(z)))]$
$\mathcal{L}_{G}^{\mathrm{ns}} = -\mathbb{E}_{z}[\log D(G(z))]$
$D^{*}(x) = \frac{p_{r}(x)}{p_{r}(x) + p_{g}(x)}$
$V(D^{*}, G) = 2\,\mathrm{JSD}(p_{r} \,\|\, p_{g}) - \log 4$

Where:

$V(D,G)$ is the value function; the discriminator maximizes it and the generator minimizes it, so $\mathcal{L}_{D} = -V$ is the batch-level binary cross-entropy actually coded up.
$x \sim p_{r}$ is a real sample and $D(x) \in [0,1]$ is the estimated probability that its input is real.
$z \sim p_{z}$ is a latent drawn from a fixed prior, usually $\mathcal{N}(0, I)$ ; $G(z)$ is the generated sample and $p_{g}$ the distribution it induces.
$\mathcal{L}_{G}^{\mathrm{sat}}$ is the literal minimax generator loss and $\mathcal{L}_{G}^{\mathrm{ns}}$ the non-saturating heuristic; they share the same optimum but differ in gradient scale by a factor of $(1 - D)/D$ .
$D^{*}$ is the optimal discriminator for a fixed $G$ , obtained by maximizing the integrand pointwise; it equals $1/2$ everywhere when $p_{g} = p_{r}$ .
$\mathrm{JSD}$ is bounded in $[0, \log 2]$ , so $V(D^{*},G)$ ranges over $[-\log 4, 0]$ and hits its global minimum $-\log 4 \approx -1.386$ only at $p_{g} = p_{r}$ .
Practical condition: $D^{*}$ is never reached, since the discriminator takes $k$ steps (typically $k = 1$ ) per generator step, and the whole derivation assumes optimization in function space rather than over network parameters.

The main alternatives keep the same alternating structure and change only the per-sample penalty. LSGAN replaces the log with a squared error, which pushes fake samples toward the decision boundary and stops the loss from flattening, at the cost of also penalizing samples that are already comfortably on the correct side. WGAN drops the sigmoid entirely: the critic maximizes $\mathbb{E}[D(x)] - \mathbb{E}[D(G(z))]$ under a 1-Lipschitz constraint, enforced in WGAN-GP by a two-sided penalty on the gradient norm at interpolated points, and the generator simply maximizes $\mathbb{E}[D(G(z))]$ . Hinge loss, the default in BigGAN and many diffusion-era discriminators, uses $\max(0, 1 - D(x))$ on reals and $\max(0, 1 + D(G(z)))$ on fakes, so a correctly classified sample beyond the margin contributes exactly zero gradient and the discriminator stops overfitting easy examples. Independently of the family chosen, the R1 penalty $\frac{\gamma}{2}\mathbb{E}_{x \sim p_{r}}[\|\nabla_{x} D(x)\|^{2}]$ is what turns the local dynamics from oscillatory into convergent.

Line chart of the discriminator loss contributed by a single fake sample as a function of its logit, comparing the smooth softplus curve of binary cross-entropy, the piecewise linear hinge loss that becomes exactly zero below minus one, and the unbounded linear Wasserstein critic loss

Figure 2: Loss contributed by one fake sample as a function of its logit $t$ . BCE uses $\log(1 + e^{t})$ and decays smoothly but never quite to zero, hinge is exactly flat once the sample clears the $-1$ margin, and the Wasserstein critic stays linear so it never saturates and must be constrained by a Lipschitz penalty.

Property	Non-saturating BCE	Hinge	WGAN-GP
Discriminator term on a fake	softplus of the logit, smooth and always positive	max(0, 1 + logit), exactly zero past the margin	the raw critic score, unbounded and linear
Generator term	minus log of the fake probability	minus the critic score, no hinge applied	minus the critic score
Behavior when D wins	gradient shrinks but stays usable via the non-saturating form	D stops learning from easy samples, which limits overfitting	score gap keeps growing, so the generator still gets signal
Extra cost per step	none beyond an optional R1 penalty, often lazily applied every 16 steps	same as BCE	a double backward pass for the penalty plus roughly 5 critic steps per generator step
Typical use	StyleGAN family with R1, and most adversarial auxiliary losses	BigGAN, SAGAN, and many feature-space discriminators	settings where a meaningful loss-versus-quality correlation is wanted

August 16, 2026

DL0100 GAN Framework

Explain the GAN framework: what are the two networks optimizing, and why does the generator learn anything at all?

Answer

A Generative Adversarial Network trains two networks against each other. The generator $G$ maps a latent noise vector $z \sim p_z$ (typically $\mathcal{N}(0, I)$ ) to a sample in data space, and the discriminator $D$ outputs the probability that its input came from the real dataset rather than from $G$ . They share one value function in a two-player minimax game: $D$ maximizes the log-likelihood of correctly classifying real and fake batches, while $G$ minimizes the same quantity, so the generator’s only learning signal is the gradient that flows backward through the discriminator into the fake samples. The theory is clean: for a fixed $G$ the optimal discriminator is the density ratio $p_{data}/(p_{data} + p_g)$ , and substituting it turns the generator’s objective into the Jensen-Shannon divergence between the data distribution and the model distribution, uniquely minimized when $p_g = p_{data}$ . The practical payoff is that $G$ is an implicit model: it never evaluates a likelihood, it just produces a sample in one forward pass, which is why GANs remain attractive whenever sampling latency matters.

(1) Two Networks One Objective: there is a single value function $V(D,G)$ ; the discriminator ascends it and the generator descends it, so no explicit reconstruction or likelihood term is ever written down.
(2) Discriminator As A Density Ratio: the optimal $D^*$ encodes $p_{data}(x)/p_g(x)$ , which is exactly the information the generator needs about where it is over- or under-producing mass.
(3) Implicit Sampling Model: $p_g$ is defined only through the pushforward of $p_z$ by $G$ , so sampling is a single forward pass but density evaluation is impossible.
(4) Non-Saturating Generator Loss: the theoretical $\log(1 - D(G(z)))$ term has almost no gradient while the generator is bad, so implementations maximize $\log D(G(z))$ instead.
(5) Alternating Updates, No Loss Curve To Read: training alternates discriminator and generator steps and seeks a saddle point, not a minimum, so a falling loss means nothing and quality is judged with FID or human inspection.

Diagram of the GAN framework: latent noise feeds the generator which produces fake samples, real samples and fake samples both feed the discriminator which outputs a probability of being real, and a dashed path carries the generator gradient back from the discriminator output into the generator

Figure 1: The generator never sees a real sample directly; it only receives gradient that has been routed backward through the discriminator, which is why the quality of $D$ bounds what $G$ can learn.

A training step draws a minibatch of real samples and a minibatch of latents, updates $D$ on both, then updates $G$ with $D$ held fixed. The original paper allowed $k$ discriminator steps per generator step; almost everyone now uses $k = 1$ and controls the balance with regularization instead. The three interventions that matter most in practice are spectral normalization or an R1 gradient penalty to keep the discriminator smooth, two-timescale learning rates so the discriminator can stay slightly ahead, and an exponential moving average of the generator weights for evaluation. The classic failure modes are mode collapse, where $G$ concentrates on a few outputs that currently fool $D$ , and vanishing generator gradient, where a discriminator with near-perfect separation returns almost nothing useful.

Mathematical Formulation:
$\min_G \max_D V(D, G)$
$V(D,G) = \mathbb{E}_{x \sim p_{data}}[\log D(x)]$
$\quad + \mathbb{E}_{z \sim p_z}[\log (1 - D(G(z)))]$
$D^*(x) = \frac{p_{data}(x)}{p_{data}(x) + p_g(x)}$
$C(G) = 2 \, \mathrm{JSD}(p_{data} \, \| \, p_g) - \log 4$
$\mathcal{L}_G = -\mathbb{E}_{z \sim p_z}[\log D(G(z))]$

Where:

$V(D,G)$ is the shared value function, maximized by $D$ and minimized by $G$ .
$x \sim p_{data}$ is a real sample and $z \sim p_z$ a latent vector, so $G(z)$ is a generated sample and $D(\cdot) \in (0,1)$ is the estimated probability of being real.
$p_g$ is the implicit distribution induced by pushing $p_z$ through $G$ ; it is never evaluated, only sampled.
$D^*$ is the optimal discriminator for a fixed $G$ , and $C(G) = \max_D V(D,G)$ is the resulting generator criterion; $\mathrm{JSD}$ is the Jensen-Shannon divergence, so $C(G) \geq -\log 4$ with equality only at $p_g = p_{data}$ .
$\mathcal{L}_G$ is the non-saturating generator loss actually used in code; it shares the same fixed point as the minimax form but has large gradient when $D(G(z))$ is near 0, which is the required starting condition of training.

Log-scale plot of generator gradient magnitude versus the discriminator score on fake samples, showing the saturating loss curve one over one minus D staying near one at low scores while the non-saturating curve one over D rises to one hundred

Figure 2: Gradient magnitude of the two generator losses with respect to $D(G(z))$ . At the start of training the discriminator wins easily, the saturating form contributes almost nothing, and the non-saturating form is roughly 100x larger at a score of 0.01.

Property	GAN	VAE	Diffusion
Training objective	Adversarial minimax, no likelihood term	ELBO, a lower bound on log-likelihood	Weighted denoising regression on noised inputs
Sampling cost	One forward pass through the generator	One forward pass through the decoder	20 to 1000 network evaluations unless distilled
Sample sharpness	Very sharp, no pixel-averaging term to blur output	Typically blurry from the reconstruction term	Very sharp, current state of the art on text-to-image
Mode coverage	Weakest; mode collapse is the signature failure	Good coverage, over-smoothed samples	Good coverage from a likelihood-style objective
Training stability	Saddle-point game, needs penalties and careful balance	Single stable objective	Single stable objective, scales predictably

August 16, 2026

DL0097 Variational Autoencoder VAE

What is a variational autoencoder (VAE), and how does it differ from a plain autoencoder?

Answer

A variational autoencoder is a latent-variable generative model trained by amortized variational inference, not a compression network with a smaller bottleneck. A plain autoencoder learns a deterministic map $z = f_\phi(x)$ and minimizes reconstruction error alone, so nothing constrains where codes land: the latent space is an arbitrary point cloud with holes, and decoding a random vector usually produces garbage. A VAE instead makes the encoder output the parameters of a distribution $q_\phi(z|x) = \mathcal{N}(\mu_\phi(x), \sigma_\phi^2(x))$ , samples $z$ from it, and optimizes the evidence lower bound (ELBO), which adds a KL term pulling every posterior toward a fixed prior $p(z) = \mathcal{N}(0, I)$ . That single extra term is what turns an encoder-decoder pair into a generator: after training you can draw $z \sim \mathcal{N}(0, I)$ and decode it, because the aggregate posterior now covers the prior. The sampling step is made differentiable by the reparameterization trick, $z = \mu + \sigma \odot \epsilon$ with $\epsilon \sim \mathcal{N}(0, I)$ , which keeps the stochastic node out of the gradient path.

(1) Probabilistic Encoder: the encoder emits a mean and a log-variance per latent dimension rather than a point, so each input maps to a small blob of latent space instead of a single coordinate.
(2) Two-Term Objective: the loss is reconstruction plus a KL regularizer against the prior; a plain autoencoder has only the first term, which is exactly why it is not generative.
(3) Reparameterization Trick: sampling is rewritten as a deterministic function of the parameters and an external noise draw, giving a low-variance pathwise gradient instead of a high-variance score-function estimator.
(4) Sampling From The Prior: because the KL term forces the posteriors to overlap and fill the prior, ancestral sampling ( $z \sim p(z)$ , then decode) yields plausible data, and interpolation between two codes stays on the data manifold.
(5) Explicit Likelihood Bound: the ELBO is a lower bound on $\log p_\theta(x)$ , so a VAE gives a comparable density estimate, whereas an autoencoder’s reconstruction error has no probabilistic meaning.

Two-row diagram: the top row shows a plain autoencoder mapping input x through an encoder to a single deterministic code and back through a decoder; the bottom row shows a VAE whose encoder emits mu and log sigma squared, a sampling step z equals mu plus sigma times epsilon, and a decoder, with a KL term pulling the posterior toward the standard normal prior

Figure 1: The structural difference is one node: the plain autoencoder passes a single deterministic code to the decoder, while the VAE passes a sample from a learned Gaussian that the KL term keeps anchored to $\mathcal{N}(0, I)$ .

Two practical details dominate real training runs. The first is the balance between the two loss terms: with a strong decoder or a large KL weight, the cheapest solution is to set $q_\phi(z|x) = p(z)$ and ignore the latent entirely, a failure called posterior collapse that shows up as a KL term decaying to near zero while reconstruction stalls. Standard mitigations are KL annealing (ramp the weight from 0 to 1 over the first epochs) and free bits (do not penalize a dimension until its per-dimension KL exceeds a floor of roughly 0.05 to 0.5 nats). The second is the choice of likelihood: a diagonal Gaussian decoder is equivalent to an MSE reconstruction loss, which averages over plausible outputs and is the direct cause of the blurry samples VAEs are known for; discretized logistic or categorical likelihoods sharpen results noticeably. Modern image systems exploit this honestly, using a KL-regularized autoencoder as the perceptual compressor and putting the generative burden on a diffusion model in that latent space, as in Stable Diffusion.

Mathematical Formulation:
$\log p_\theta(x) \geq \mathcal{L}(\theta, \phi; x)$
$\mathcal{L} = \mathcal{L}_{rec} - \beta \, D_{KL}(q_\phi \| p)$
$\mathcal{L}_{rec} = \mathbb{E}_{q_\phi(z|x)}[\log p_\theta(x|z)]$
$q_\phi(z|x) = \mathcal{N}(\mu_\phi(x), \sigma_\phi^2(x))$
$z = \mu_\phi(x) + \sigma_\phi(x) \odot \epsilon$
$\epsilon \sim \mathcal{N}(0, I)$
$D_{KL} = -\frac{1}{2} \sum_{j=1}^{d} \delta_j$
$\delta_j = 1 + \log \sigma_j^2 - \mu_j^2 - \sigma_j^2$

Where:

$\mathcal{L}$ is the ELBO, a lower bound on the marginal log-likelihood $\log p_\theta(x)$ ; the gap between them equals $D_{KL}(q_\phi(z|x) \| p_\theta(z|x))$ , so maximizing the bound both fits the data and sharpens the approximate posterior.
$x$ is the observation and $z \in \mathbb{R}^{d}$ the latent code, with $d \ll \dim(x)$ in the usual bottleneck setting.
$q_\phi(z|x)$ is the encoder (recognition model) with parameters $\phi$ , and $p_\theta(x|z)$ is the decoder (likelihood) with parameters $\theta$ .
$p(z) = \mathcal{N}(0, I)$ is the fixed prior, and $j \in \{1, \ldots, d\}$ indexes latent dimensions in the closed-form KL, where $\mu_j$ and $\sigma_j$ are the encoder outputs for example $x$ .
$\odot$ is elementwise multiplication and $\epsilon$ is the external noise draw that makes the sample differentiable in $\mu$ and $\sigma$ .
$\beta$ is the KL weight; $\beta = 1$ recovers the exact ELBO, $\beta > 1$ gives the beta-VAE disentanglement regime, and $\beta = 0$ degenerates to a plain autoencoder. Networks are typically initialized so $\log \sigma_j^2 \approx 0$ , which starts training near the prior.

Two scatter panels of a two-dimensional latent space: on the left the plain autoencoder codes sit in six distant clusters far from the origin with large empty gaps and prior samples falling in a hole, on the right the VAE codes form overlapping blobs filling the unit and two-sigma circles of the standard normal prior so prior samples land on data

Figure 2: Encoded training data in a 2D latent space. The plain autoencoder is free to scatter codes anywhere, so a draw from $\mathcal{N}(0, I)$ lands in a hole; the KL term compresses the VAE’s aggregate posterior onto the prior, so the same draw hits populated territory.

Property	Plain Autoencoder	VAE	VQ-VAE
Encoder output	One deterministic code vector per input	Distribution parameters, mean and log-variance	Continuous vector snapped to the nearest codebook entry
Training objective	Reconstruction error only	ELBO: reconstruction minus beta times KL to N(0, I)	Reconstruction plus codebook and commitment losses, no KL
Latent geometry	Arbitrary scale, holes and gaps between clusters	Smooth and prior-matched, interpolation stays on-manifold	Discrete grid of K entries, no notion of interpolation
Generating new data	Not supported, a random code decodes to noise	Draw z from the prior and decode, one forward pass	Needs a learned prior over codes, such as a transformer
Typical failure	Memorizes an identity map when the bottleneck is wide	Posterior collapse, blurry Gaussian-likelihood samples	Codebook collapse with most entries unused
Common use	Denoising, compression, anomaly detection	Generative modeling and the latent space of latent diffusion	Discrete tokens for autoregressive image and audio models

August 16, 2026

DL0070 Multi-Task Loss Balancing

How would you design a loss for multi-task learning when the tasks have very different scales?

Answer

Start from the failure mode: with a naive sum $L = \sum_i L_i$ , each task contributes gradient in proportion to its loss scale, so a depth-regression MSE sitting near 200 drowns out a segmentation cross-entropy sitting near 1, and the shared trunk optimizes the loud task while the quiet ones stall. The fix comes in three tiers. First, put the losses on a common scale by dividing each by a fixed or running estimate of its magnitude, such as its initial value. Second, learn the weights: Kendall et al.’s uncertainty weighting attaches a trainable $\sigma_i$ per task and minimizes $\sum_i L_i / (2\sigma_i^2) + \log \sigma_i$ , derived from Gaussian and categorical likelihoods, so noisy tasks are down-weighted automatically while the $\log \sigma_i$ term stops $\sigma_i$ from blowing up. Third, when scales are balanced but gradient directions still conflict, move to gradient-level methods: GradNorm tunes the task weights so each task’s gradient norm on the shared trunk approaches a common scale adjusted by its relative inverse training rate, so tasks that are learning slowly get pushed harder, and PCGrad projects away mutually conflicting components.

(1) Scale Equals Loudness: task $i$ ‘s gradient share scales with the units of $L_i$ , so the unweighted sum is an implicit weighting set by arbitrary unit choices; expressing depth in millimeters instead of meters multiplies its MSE by $10^6$ and hands it the entire gradient budget.
(2) Uncertainty Weighting: treat $\sigma_i$ as task $i$ ‘s observation noise; the weight $1/(2\sigma_i^2)$ falls as noise grows while the $\log \sigma_i$ penalty (the likelihood’s normalizing constant) rises, so the optimum is a genuine trade-off learned by gradient descent alongside the network weights.
(3) Magnitude Is Not Direction: scale balancing fixes how loudly tasks speak, not whether they agree; when task gradients point in opposing directions (negative transfer), gradient-space methods such as GradNorm or PCGrad are the right lever.

Grouped bar chart on a log scale of illustrative gradient norms on the shared trunk for segmentation, depth, and normals tasks: unweighted bars are 1.1, 160, and 0.8, while uncertainty-weighted bars are 0.9, 1.0, and 0.7

Figure 1: Illustrative three-task trunk, gradient norms on a log scale: unweighted, the depth MSE contributes over 100x the others; once each loss carries a learned $1/(2\sigma_i^2)$ weight, the three contribute comparably.

The uncertainty objective is not ad hoc. Modeling regression noise as $p(y \mid f(x)) = \mathcal{N}(f(x), \sigma^2)$ gives the negative log-likelihood $\|y - f(x)\|^2 / (2\sigma^2) + \log \sigma$ per task, up to constants; classification slots in through a scaled softmax likelihood whose approximation yields the analogous weight $1/\sigma^2$ (without the factor 2) alongside the same $\log \sigma$ penalty. Two properties matter in practice: $\sigma_i$ is learned by the same optimizer as the network, so balancing needs no manual grid search; and the $\log \sigma_i$ term keeps the objective honest, because without it every $\sigma_i$ would grow without bound, all task weights would collapse to zero, and nothing would be learned. Most implementations optimize $s_i = \log \sigma_i^2$ and compute $e^{-s_i} L_i + s_i$ for numerical stability.

Two panels against task uncertainty sigma: top shows the task weight 1/(2 sigma^2) collapsing on a log scale as sigma grows from 0.25 to 4; bottom shows the penalty log sigma rising over the same range

Figure 2: The learned trade-off: as a task’s noise $\sigma$ grows, its weight $1/(2\sigma^2)$ collapses while the penalty $\log \sigma$ rises, so the optimizer cannot silence a noisy task for free.

Mathematical Formulation:
$L_{\mathrm{naive}} = \sum_{i=1}^{T} L_i$
$L_{\mathrm{total}} = \sum_{i=1}^{T} \left( \frac{1}{2\sigma_i^2}\, L_i + \log \sigma_i \right)$

Where:

$L_i$ is the loss of task $i$ (cross-entropy for segmentation, MSE for depth) and $T$ the number of tasks sharing the trunk.
$\sigma_i$ is a learned per-task scalar modeling homoscedastic (task-level, input-independent) observation noise, initialized at 1 and trained by the same optimizer as the weights.
$1/(2\sigma_i^2)$ is the derived task weight and $\log \sigma_i$ the likelihood’s normalizing term that penalizes inflating $\sigma_i$ ; in code, parameterize $s_i = \log \sigma_i^2$ and optimize $e^{-s_i} L_i + s_i$ instead.

Method	What It Balances	Extra Cost	When to Reach for It
Fixed Weights (Grid Search)	Loss scales, set by hand	Search cost grows fast with task count	Two tasks and plenty of compute
Loss Normalization	Loss magnitudes via running scale estimates	Negligible	Quick baseline before anything fancier
Uncertainty Weighting (Kendall)	Loss weights via learned sigma per task	One extra scalar per task	Default starting point for shared-trunk training
GradNorm	Gradient norms toward a common scale by training rate	Extra backward bookkeeping each step	Tasks learning at very different speeds
PCGrad	Conflicting gradient directions	Per-task gradients every step	Negative transfer between tasks

August 12, 2026

DL0047 Focal Loss II

Please compare focal loss and weighted cross-entropy.

Answer

Weighted cross-entropy (WCE) multiplies each class’s loss by a fixed weight $\alpha_t$ : it corrects class frequency but treats every sample of a class identically. Focal loss (FL) adds a per-sample, difficulty-dependent factor $(1 - p_t)^\gamma$ : it corrects prediction difficulty, automatically ignoring easy examples regardless of class. WCE is simple and robust; FL is stronger when an ocean of easy negatives overwhelms learning, but it can amplify noisy labels.

(1) What Each Balances: WCE reweights by class prior; FL reweights by per-sample hardness (and can include $\alpha_t$ too).
(2) Gradient Behavior: WCE still lets millions of correctly-classified negatives contribute steady gradient; FL drives their contribution toward zero.
(3) Robustness: WCE has one interpretable hyperparameter and is safe with noisy labels; FL’s focus on hard examples can overfit label noise and needs tuned $\gamma, \alpha_t$ .

Mathematical Formulation:
$\mathrm{WCE}(p_t) = -\alpha_t \log(p_t)$
$\mathrm{FL}(p_t) = -\alpha_t (1 - p_t)^\gamma \log(p_t)$

Where:

$p_t$ is the model’s predicted probability for the ground-truth class; $\alpha_t$ is the fixed per-class weight.
$\gamma \geq 0$ is the focusing parameter, the entire difference between the two losses; at $\gamma = 0$ focal loss collapses to WCE.

Property	Weighted Cross-Entropy	Focal Loss
Handles class imbalance	Yes: fixed class weights	Yes: class weight + hardness factor
Focuses on hard samples	No: easy samples still dominate	Yes: easy samples fade to zero
Hyperparameters	One: $\alpha_t$	Two: $\gamma$ and $\alpha_t$
Main risk	Underwhelming at extreme imbalance	Overfits noisy / mislabeled hard examples

Table 1: WCE vs focal loss: the same class weighting, but only focal loss adds per-sample difficulty weighting.

Curves of cross-entropy, weighted cross-entropy at two alpha values, and focal loss at several gamma values versus true-class probability, showing WCE as a uniform vertical rescale while focal loss bends the curve down at high probability.

Figure 1: WCE rescales the CE curve uniformly (same shape, different height); focal loss bends the shape, crushing the high- $p_t$ (easy) region toward zero.

Rule of Thumb: Moderate imbalance (up to ~10:1) → WCE is enough; extreme imbalance with floods of easy negatives (dense detection, 1000:1) → focal loss; noisy labels → prefer WCE.

October 27, 2025

DL0046 Focal Loss
What is focal loss, and why does it help with class imbalance?
Answer
Focal loss augments cross-entropy with a modulating factor $(1 - p_t)^\gamma$ that shrinks the loss of easy, well-classified examples toward zero, so the vast number of easy negatives in imbalanced data stops dominating the gradient. Training then concentrates on hard, misclassified, or minority-class examples, which is where the useful learning signal actually lives.
(1) Modulating Factor: $(1 - p_t)^\gamma$ is near 0 when the model is confident and correct, and near 1 when it is wrong, so easy examples are automatically down-weighted.
(2) Class Imbalance Effect: In detection-style problems with ~100k easy negatives per positive, plain cross-entropy is swamped by their accumulated gradient; focal loss suppresses it.
(3) Optional Balancing: A per-class weight $\alpha_t$ can additionally correct the base-rate skew of the classes themselves.
Mathematical Formulation:
$\mathrm{FL}(p_t) = -\alpha_t\, (1 - p_t)^\gamma \log(p_t)$
Where:
$p_t$ is the model’s predicted probability for the ground-truth class.
$\gamma \geq 0$ is the focusing parameter: larger values suppress easy examples harder; $\gamma = 0$ with $\alpha_t = 1$ recovers plain cross-entropy.
$\alpha_t \in (0,1)$ is an optional per-class balancing weight for class $t$ .

Figure 1: Focal loss curves for several $\gamma$ : as $\gamma$ grows, the loss of confident correct predictions (high $p_t$ ) collapses toward zero while hard examples keep full weight.
Scenario $p_t$ (True-Class Prob) Factor $(1-p_t)^\gamma$ Effect on Loss
Easy example High Low → 0 Down-weighted to near zero
Hard example Low High → 1 Keeps full learning signal
Table 1: How the modulating factor behaves: easy examples fade out, hard examples dominate the gradient.
Typical Settings: The RetinaNet paper found $\gamma = 2$ with $\alpha = 0.25$ (for the positive class) works well; the useful range is usually $\gamma \in [1, 3]$ ; too large over-focuses on noisy or mislabeled outliers.
Login to view more content
October 27, 2025
ML0053 Hinge Loss for SVM
Explain the Hinge Loss function used in SVM.
Answer
The hinge loss is the key element of Support Vector Machines: it penalizes both misclassified points and correctly classified points that lie inside the margin. Points classified correctly with margin to spare get zero loss; the loss grows linearly as a point moves toward or across the decision boundary. This structure is exactly what pushes the SVM to maximize the margin, promoting robust, generalizable boundaries.
(1) Zero Loss Zone: When $y \cdot f(\mathbf{x}) \geq 1$ , the point is correctly classified and outside (or exactly on) the margin, the loss is 0.
(2) Linear Penalty Zone: When $y \cdot f(\mathbf{x})$ falls below 1, i.e., inside the margin or misclassified, the loss grows linearly with the violation.
(3) Convex But Not Smooth: The kink at $y \cdot f(\mathbf{x}) = 1$ makes the function non-differentiable there, so optimization uses subgradients instead of plain gradients.

Figure 1: Hinge loss vs the margin $y \cdot f(\mathbf{x})$ : flat at zero once the point is beyond the margin (right of the dashed line at 1), ramping up linearly inside the margin and for misclassified points (left of it). The kink at 1 is where subgradients take over.
Mathematical Formulation:
$\text{Hinge Loss} = \max\big(0,\; 1 - y \cdot f(\mathbf{x})\big)$
Where:
$y \in \{-1, +1\}$ is the true label.
$f(\mathbf{x})$ is the raw model output (the signed score, before any threshold).
The product $y \cdot f(\mathbf{x})$ is the (functional) margin: positive means correctly classified, ≥ 1 means correct with margin.
Login to view more content
June 11, 2025

Scenario	$p_t$ (True-Class Prob)	Factor $(1-p_t)^\gamma$	Effect on Loss
Easy example	High	Low → 0	Down-weighted to near zero
Hard example	Low	High → 1	Keeps full learning signal