DL0075 Attention Sink

An attention sink is a token that a head dumps its attention mass onto when it has nothing relevant to retrieve, and in decoder-only LLMs that token is almost always the first one in the sequence (often the BOS token). Measured on a trained model, a middle-layer head can place well over half of its probability on that single position even though its value vector carries almost no task-relevant content. The cause is structural rather than semantic: softmax normalizes every attention row to sum to exactly 1, so a head has no way to say “attend to nothing” and must put the leftover mass somewhere. Causal masking makes the first token the only key visible from every query position, and it is the one token whose representation is not yet mixed with context, so training converges on it as the default no-op target. The model then reinforces the arrangement by growing massive activations and unusually large key norms at that position, which is why the sink shows up as an outlier in both attention maps and activation histograms.

(1) Softmax Has No No-Op: the row constraint means an idle head still emits a full distribution, so the mass must land on some low-cost token.
(2) Why The First Token Wins: under causal masking it is the only key every query can see, and it precedes all content, so it is both universally available and information-free, the ideal dumping ground.
(3) The Sink Is Load-Bearing: it is not decorative; drop the first token from a sliding-window KV cache and perplexity jumps by one to two orders of magnitude, because every head loses its default target and redistributes mass onto real tokens.
(4) Fingerprints Beyond Attention: the same position carries hidden-state outliers that widen activation ranges, which is a known headache for per-tensor quantization; vision transformers show the analogous effect by repurposing background patches as sinks.

Figure 1: Illustrative single head at query position 512: the first token absorbs 0.42 of the mass, the most recent 64 positions share 0.25, and the roughly 444 middle positions split only 0.22, about each.

The cleanest way to read the sink is as a learned bias term smuggled into the softmax. Because the denominator sums over all visible keys, the only way for a head to shrink the weight it gives to real tokens is to inflate the score of some other key; the first token, whose value vector contributes near-zero to the output, is that escape valve. This also explains the standard fixes. Giving the softmax an explicit extra term in the denominator (the off-by-one softmax, or a learned per-head sink logit as used in some recent open-weight models such as OpenAI’s gpt-oss) lets attention rows sum to less than 1, so no real token needs to be sacrificed. Prepending dedicated register tokens achieves the same thing with ordinary softmax by supplying a scratch position, which is what cleaned up the noisy attention maps in vision transformers. Both changes must be present during pretraining: retrofitting them onto a model that already routes mass through token 0 changes the normalization the weights were trained under.

Mathematical Formulation:

Where:

• is the scaled dot-product score between query and key , with head dimension .
• is the attention weight; indexes query positions and index key positions, restricted to by causal masking, which is why is the only key shared by every query.
• The third line is the normalization constraint that creates the sink: no configuration of scores lets a head emit less than one unit of total mass.
• is a learned per-head sink logit added to the denominator, so can fall below 1; the off-by-one softmax is the special case , since .

Figure 2: Illustrative streaming behavior: a plain sliding window collapses the moment the first token leaves the KV cache, while retaining a handful of sink tokens alongside the window keeps perplexity flat far beyond the pretraining length.