DL0046 Focal Loss

Focal loss augments cross-entropy with a modulating factor that shrinks the loss of easy, well-classified examples toward zero, so the vast number of easy negatives in imbalanced data stops dominating the gradient. Training then concentrates on hard, misclassified, or minority-class examples, which is where the useful learning signal actually lives.

(1) Modulating Factor: is near 0 when the model is confident and correct, and near 1 when it is wrong, so easy examples are automatically down-weighted.
(2) Class Imbalance Effect: In detection-style problems with ~100k easy negatives per positive, plain cross-entropy is swamped by their accumulated gradient; focal loss suppresses it.
(3) Optional Balancing: A per-class weight can additionally correct the base-rate skew of the classes themselves.

Mathematical Formulation:

Where:

• is the model’s predicted probability for the ground-truth class.
• is the focusing parameter: larger values suppress easy examples harder; with recovers plain cross-entropy.
• is an optional per-class balancing weight for class .

Figure 1: Focal loss curves for several : as grows, the loss of confident correct predictions (high ) collapses toward zero while hard examples keep full weight.

Scenario	(True-Class Prob)	Factor	Effect on Loss
Easy example	High	Low → 0	Down-weighted to near zero
Hard example	Low	High → 1	Keeps full learning signal

Table 1: How the modulating factor behaves: easy examples fade out, hard examples dominate the gradient.

Typical Settings: The RetinaNet paper found with (for the positive class) works well; the useful range is usually ; too large over-focuses on noisy or mislabeled outliers.