DL0082 LLM Temperature

Temperature is a single positive scalar that divides the model’s logits before the softmax at every decoding step, so it reshapes the next-token distribution without touching the model weights. At you sample from exactly the distribution the model was trained to produce; values below 1 sharpen that distribution toward the highest-scoring token, and values above 1 flatten it toward uniform over the vocabulary. The transformation is monotone in the logits, so temperature never changes which token is ranked first, only how much probability the gaps between logits are worth. The practical effect is a variance knob: low temperature gives repetitive but stable text, high temperature gives diverse text with a much fatter tail of implausible tokens. Because it is applied independently at every step of an autoregressive loop, a small per-token change in tail mass compounds across a long generation.

(1) Logit Rescaling, Not Reranking: dividing all logits by the same preserves their order, so the argmax token is invariant and only the sampling probabilities move.
(2) The Two Limits: as approaches 0 sampling collapses to greedy decoding, and as grows the distribution approaches uniform; frameworks special-case as argmax because the division itself is undefined.
(3) Tails Gain Disproportionately: temperature is a power transform of the base distribution, so unlikely tokens change by the largest relative factor; with logits the top token falls from 0.644 to 0.455 at while the weakest rises from 0.032 to 0.102.
(4) Compounding Across Steps: the model conditions on its own samples, so one low-quality token drawn from the inflated tail can steer every subsequent token, which is why high temperature degrades long generations faster than short ones.
(5) Not A Truthfulness Knob: lowering reduces variance, not error; if the mode is wrong, greedy decoding returns the wrong answer with total confidence.

Figure 1: The same four logits under three temperatures. Entropy grows from 0.66 bits at to 1.80 bits at , and the weakest token gains roughly 47x probability across that range while the top token loses less than half.

Temperature never acts alone in a real decoder. The usual pipeline is logits → temperature → top-k or top-p truncation → renormalize → sample, and the ordering matters: because temperature runs first, raising it inflates the tail that nucleus sampling then has to cut, so the same threshold admits a larger candidate set than it did at . That is why “high temperature plus top-p” is not a safety net, and why many production defaults pair a moderate temperature with a fixed truncation instead of pushing either one hard. A separate but easily confused use of the same formula is temperature scaling for calibration, where a single is fit on held-out data to correct an overconfident classifier; there the goal is matching predicted confidence to observed accuracy, not generating diverse text, and the fitted value is a property of the model rather than a user-facing creativity dial.

Mathematical Formulation:

Where:

• is the sampling probability of vocabulary token at temperature , and is that token’s raw logit from the final linear layer.
• is the vocabulary size and indexes every candidate in the softmax denominator, so all probabilities are renormalized after rescaling.
• is the temperature: values below 1 sharpen the distribution, values above 1 flatten it, and is an identity operation.
• The second line shows temperature is a power transform with exponent applied to the base distribution, which is the compact reason tail tokens move by the largest relative factor.
• is the argmax token, and the fourth line is the greedy limit; implementations therefore branch to argmax at instead of evaluating the ratio.
• is the entropy of the next-token distribution in bits; it increases monotonically with and is bounded above by .

Figure 2: Sweeping over the same four logits. Entropy is monotonically increasing and saturates at bits, while the tail mass of the two weakest tokens more than doubles between and .