DL0050 Knowledge Distillation

Knowledge distillation (KD) trains a small student model to imitate a large, accurate teacher model. The key trick is learning from the teacher’s temperature-softened output distribution (“dark knowledge”: e.g., a cat image looks a bit like a dog, nothing like a truck) rather than only from one-hot hard labels. The student ends up much smaller and faster while retaining most of the teacher’s accuracy, which is why KD is the standard route to deployable models.

(1) Soft Targets: Teacher logits are passed through softmax with a temperature , exposing inter-class similarity structure that hard labels hide.
(2) Combined Loss: The student minimizes a mix of distillation loss (KL to the teacher’s soft targets) and ordinary cross-entropy on the true labels.
(3) Benefits: Compression and latency for edge/real-time deployment, plus a regularization effect: students often generalize better than the same architecture trained on hard labels alone.

Mathematical Formulation:

Where:

• is the logit for class , the number of classes, and the temperature; higher yields a smoother distribution.
• balances hard-label CE against distillation; the factor rescales the KL term because softening shrinks its gradients by .

Figure 1: Temperature smoothing: at only the winner class is visible; at higher the class-similarity ratios (“dark knowledge”) emerge.

Training Setup: The teacher runs in inference mode (frozen); only the student’s weights update. Both models see the same inputs, and the two losses are computed on the student’s outputs only.

Figure 2: The KD setup: the student learns from both the teacher’s soft targets (distillation loss) and the ground truth (student loss).

Practical Caveats: A weak or biased teacher transfers its errors; and need tuning; and an extremely small student may lack the capacity to absorb the teacher. Intermediate-feature distillation and task-specific data help close the gap.