DL0169 Softmax-1 Off-by-One Attention

Written by

What is the Softmax-1 (off-by-one) modification to attention, and why does subtracting one from the denominator improve length generalization and register-token behavior in recent Transformer architectures?

Answer

Softmax-1, also called off-by-one or quiet attention, replaces the attention normalizer $\sum_j e^{z_j}$ with $1 + \sum_j e^{z_j}$ . That extra constant is exactly what you get by appending a virtual key with logit 0 and value vector 0, so the weights over real tokens now sum to $S/(1+S)$ , which is strictly below one, and a head that finds nothing relevant can emit the zero vector. Ordinary softmax has no such option, because its weights are forced to sum to one whether or not any key matches. Heads that want to be a no-op therefore learn to dump their leftover mass on a low-information token, usually the first token or a delimiter, and to inflate that token’s residual to enormous magnitude: the massive activations that dominate activation-quantization error and the artifact tokens that pollute ViT attention maps. The name refers to the denominator being off by one; the operational reading is that one unit of attention budget is subtracted from the real keys and parked in a null option that costs nothing to select.

(1) One Constant, No New Parameters: adding 1 to the denominator is algebraically identical to concatenating a zero-logit, zero-value key, so the change touches one line of the kernel and adds no weights.
(2) Mass Below One Is The Whole Point: the head gains an explicit “abstain” action, and the output norm $\lVert o \rVert$ can go to zero without any token’s value vector having to be zero.
(3) Absolute Instead Of Relative Scores: ordinary softmax is invariant to a constant shift of all logits, so “nothing is similar” is inexpressible; the fixed zero reference makes the logit level itself meaningful.
(4) The Sink Becomes A Mechanism, Not A Token: vanilla models route no-op mass through a specific KV slot that a sliding window may evict, whereas the constant is always present, which is what stabilizes long-context and windowed decoding.
(5) Registers Stop Being Garbage Dumps: once abstention is free, register and background tokens keep their global-storage role without absorbing surplus attention, so their norms stay moderate and attention maps stay readable.
(6) Production Form Is A Learned Sink Logit: replacing the 1 with $e^{s}$ for a per-head learned $s$ strictly generalizes Softmax-1, which is the $s = 0$ special case.

Two-panel grouped bar chart of attention weights over six keys plus a null slot. In panel A every logit is below zero and ordinary softmax still renormalizes to sum one, while softmax-1 keeps only 0.17 of the budget and sends 0.83 to the null slot. In panel B one logit is six and the two methods are nearly identical, with the null slot receiving only a quarter of a percent.

Figure 1: The gate is selective, not a uniform rescaling. When no key matches (panel A), ordinary softmax renormalizes a row of weak scores into a confident-looking distribution, while Softmax-1 keeps 17% of the budget on real keys and abstains with the rest. When a real match exists (panel B), the constant 1 is negligible against $e^{6} \approx 403$ and the two are indistinguishable, so the modification only fires where the head has nothing to say.

Mathematical Formulation:
$\mathrm{softmax}(z)_i = \frac{e^{z_i}}{\sum_{j=1}^{L} e^{z_j}}$
$\mathrm{softmax}_1(z)_i = \frac{e^{z_i}}{1 + \sum_{j=1}^{L} e^{z_j}}$
$S = \sum_{j=1}^{L} e^{z_j}$
$m = \frac{S}{1 + S}$
$m = \sigma(z + \log L)$
$p_i = \frac{e^{z_i}}{e^{s} + \sum_{j=1}^{L} e^{z_j}}$
$o = \sum_{i=1}^{L} p_i v_i$

Where:

$z_i = q^{\top} k_i / \sqrt{d}$ is the scaled dot-product logit for key $i$ , and $L$ is the number of visible keys after masking.
$S$ is the ordinary denominator, so the added 1 is just $e^{0}$ : the virtual key’s exponentiated logit.
$m$ is the total attention mass retained by real keys; the complement $1/(1+S)$ is the abstention share, and it is positive for every finite logit row.
The uniform-logit identity uses $\sigma$ for the logistic function: with all $L$ logits tied at $z$ , the mass is a logistic gate whose half-way point sits at $z = -\log L$ .
$s$ is a learned per-head sink logit replacing the constant, and $s = 0$ recovers Softmax-1 exactly.
$o$ is the head output; because $\sum_i p_i$ can be near zero, $o \approx 0$ is reachable without constraining any $v_i$ .

The length-generalization argument has two halves. First, in ordinary softmax a no-op head’s surplus mass is $1 - m_{\mathrm{rel}}$ , and where it lands depends on how many irrelevant keys exist, so a head calibrated at a 4k training length re-partitions its budget when the context reaches 32k; the model compensates by pinning a token whose residual magnitude was tuned at the training length, and Sun et al. measured such coordinates in the thousands while the median activation sits near 0.1. Second, that sink is a KV slot, not an architectural constant. StreamingLLM showed the consequence directly: evict the first few tokens from a sliding window and perplexity explodes, while pinning four sink tokens keeps generation stable out to millions of tokens. Softmax-1 turns the sink into a term of the denominator that no eviction policy can delete, which is why windowed and extrapolated decoding stop depending on cache bookkeeping. The honest caveat is visible in the uniform-logit identity above: the abstention gate is absolute but its threshold still drifts as $-\log L$ , so pushing a head to stay quiet across a 256x context increase costs about 5.5 nats of logit headroom, which is precisely why deployed variants learn $s$ per head rather than freezing it at zero.

Line chart of total attention mass on real keys versus a uniform attention logit, for context lengths 512, 8192 and 131072. Each softmax-1 curve is a logistic whose half-way point sits at minus log L, marked by dotted vertical lines, while a flat dash-dotted line at mass one shows that ordinary softmax retains the whole budget regardless of logit level or length.

Figure 2: Softmax-1 converts the normalizer into a logistic gate on the absolute logit level, whereas ordinary softmax pins the mass at one for every logit row and every length. The gate’s half-way point moves left by $\log L$ , so a head that abstains comfortably at 512 keys leaks over half its budget at 131k unless it lowers its logits by about 5.5 nats, which is the argument for a learned sink logit rather than a hard-coded 1.

The register story is the same pressure seen in vision. Darcet et al. found that DINOv2, CLIP, and DeiT-III spontaneously repurpose a small fraction of low-information background patches as global scratch space, giving those patches norms roughly an order of magnitude above their neighbours and visibly corrupting attention maps and dense-prediction features; adding explicit register tokens removed the artifacts. Registers supply somewhere to write, while Softmax-1 supplies permission not to write, and the two are complementary rather than competing. Bondarenko et al. reached the identical diagnosis from the quantization side, naming the problem “helping attention heads do nothing” and reporting that suppressing the outliers is what makes per-tensor INT8 activation quantization viable. Reported perplexity gains from the off-by-one change alone are small and setup-dependent, so the case for it rests on outlier suppression and long-context robustness, not on loss curves.

Property	Ordinary softmax	Softmax-1	Learned sink logit	Register tokens
How a head does nothing	Dumps mass on a learned sink token and shrinks its value contribution	Pushes every logit below zero and abstains	Same, with the abstention threshold trained per head	Attends to a dedicated token reserved for scratch space
Weights sum to	Exactly 1	$S/(1+S)$ , strictly below 1	$S/(e^{s}+S)$ , strictly below 1	Exactly 1, including the register slots
Added cost	None	One constant in the denominator, zero parameters	One scalar per head	Extra sequence positions, so extra KV and quadratic prefill
Massive activations	Emerge reliably; block per-tensor INT8 activation quantization	Strongly reduced when trained from scratch	Reduced, and the head can tune how quiet it stays	Confined to registers instead of random patches, not removed
Sliding window safety	Fragile: evicting the sink token collapses perplexity	Safe: the null option is architectural, nothing to pin	Safe, and adapts to the window length during training	Safe only if the registers are never evicted
Retrofit to a trained model	Baseline	Needs continued pretraining; a drop-in swap breaks calibration	Same, though $s$ can be initialized from observed sink mass	Requires retraining or at least adapter tuning of the new tokens

LLM Transformer

DL0169 Softmax-1 Off-by-One Attention

Comments

Leave a Reply Cancel reply

More posts

MSD0064 Notification Ranking System

MSD0063 Image Copyright Detection

MSD0062 Ads Ranking System

MSD0061 Nearby Place Recommendation