ML0022 Cross Entropy Loss

Written by

Explain how Cross Entropy Loss is used for a classification task.

Answer

Cross-entropy loss (log loss) measures the distance between two probability distributions: the model’s predicted class probabilities and the true distribution (a one-hot vector for labeled data). Training a classifier means minimizing this distance, which is exactly maximum-likelihood estimation of the model parameters. In the binary case the loss averages $-[y \log p + (1-y)\log(1-p)]$ ; in the multi-class case only the true class term survives the one-hot sum, reducing the per-sample loss to $-\log p_{\text{true}}$ . The logarithm is the key design choice: it penalizes confident wrong predictions severely: as the probability assigned to the true class approaches 0 the loss explodes toward infinity, giving a strong, well-behaved gradient exactly where the model errs most. It pairs naturally with sigmoid (binary) or softmax (multi-class) output layers, whose gradients under cross-entropy take the clean form $p - y$ .

(1) Definition: Negative log-likelihood of the true labels under the predicted distribution; minimizing it is maximum-likelihood training.
(2) Confidence Sensitivity: Loss grows unboundedly as the true-class probability approaches 0; confident mistakes are punished hardest.
(3) Standard Pairing: Sigmoid output + binary cross-entropy, softmax output + categorical cross-entropy; combined gradient simplifies to $p - y$ .

Cross entropy loss minus log p rising steeply as predicted probability of the true class approaches zero

Figure 1: Per-sample loss $-\log(p)$ as a function of the probability assigned to the true class: near $p = 1$ the loss is almost zero, below $p = 0.1$ it grows steeply: confident errors dominate the gradient.

Mathematical Formulation:
$\mathcal{L}_{\text{binary}} = -\frac{1}{n} \sum_{i=1}^{n} \Big[ y_i \log p_i + (1 - y_i) \log(1 - p_i) \Big]$
$\mathcal{L}_{\text{multi}} = -\frac{1}{n} \sum_{i=1}^{n} \sum_{c=1}^{C} y_{ic} \log p_{ic} = -\frac{1}{n} \sum_{i=1}^{n} \log p_{i,\text{true}}$

Where:

$n$ is the number of samples and $C$ the number of classes.
$y_i \in \{0, 1\}$ is the true binary label; $p_i$ the predicted probability of the positive class.
$y_{ic}$ is the one-hot indicator (1 if sample $i$ belongs to class $c$ ); $p_{ic}$ the predicted probability of class $c$ , so the inner sum keeps only the true-class term.