ML0028 Softmax

Softmax is the activation used in the output layer for multi-class classification: it converts a vector of raw scores (logits) into a normalized probability distribution over the classes. Each output is the exponentiated logit divided by the sum of exponentiated logits, so every output lies in and all outputs sum to exactly 1, a genuine probability distribution that can be read as the model’s confidence in each class. Softmax assumes classes are mutually exclusive (one true class per sample); for multi-label problems where an input can belong to several classes at once, per-class sigmoid outputs are used instead, since each class needs an independent probability. Paired with cross-entropy loss, softmax gives the clean gradient , which is why the combination is the standard for multi-class training.

(1) Purpose: Turn arbitrary logits into a probability distribution: outputs in , summing to 1.
(2) Amplification: Exponentiation sharpens differences: the largest logit dominates the distribution.
(3) Scope: Multi-class (mutually exclusive) problems with cross-entropy loss; multi-label problems use independent sigmoids instead.

Figure 1: Softmax in action: raw logits (top) of arbitrary scale and sign are exponentiated and normalized into a probability distribution (bottom) that sums to 1; the relative order is preserved but differences are amplified.

Mathematical Formulation:

Where:

• is the raw score (logit) for class , and the number of classes.
• The exponential makes every term positive; dividing by the total normalizes the vector to sum 1.
• is an optional temperature: a value below 1 sharpens the distribution, softens it (used in distillation and sampling).

Figure 2: Temperature controls sharpness: low temperature pushes probability mass onto the top class (hard distribution), high temperature flattens toward uniform: the same logits, three very different distributions.