DL0168 NoPE: Causal Positional Bias Without Encodings

NoPE is a decoder-only Transformer trained with no position information injected anywhere: no learned or sinusoidal absolute embedding added to the token embeddings, no RoPE rotation inside attention, and no ALiBi distance bias on the logits. The only order-dependent structure left in the network is the causal mask, and that turns out to be enough. A bidirectional self-attention layer without positional encodings is permutation equivariant, so it can only represent a bag of tokens, but causal masking destroys that symmetry because query attends over exactly keys. The size of each token’s receptive field is therefore itself a strictly monotone function of absolute position, and any head with near-uniform logits converts that count into a readable signal in the residual stream. Haviv et al. showed that absolute position can be linearly probed out of NoPE hidden states with high accuracy while perplexity stays close to models with explicit encodings, and Kazemnejad et al. proved constructively that a NoPE decoder can represent both absolute and relative position, and that it length-generalizes at least as well as RoPE or ALiBi on small-scale algorithmic tasks.

(1) Definition: NoPE removes every explicit position term and keeps only the lower-triangular attention mask, so position is an emergent property rather than an injected feature.
(2) Broken Permutation Equivariance: without a mask, for any permutation , but the causal mask satisfies , which is exactly the asymmetry the model exploits.
(3) Counting Is The Mechanism: uniform attention over an -token window puts mass on each visible key, so the output norm encodes absolute position.
(4) Absolute First, Relative Later: layer 1 materializes an absolute code, usually anchored on a dominant BOS sink, and deeper layers subtract two codes to obtain the relative offset .
(5) Resolution Decays Quadratically: the gap between adjacent positions is , so the counting code loses discriminative power long before the context window ends.
(6) Implicit Recency Bias: trained NoPE heads develop distance-decaying attention that resembles a learned relative encoding, but with no principled extrapolation knob, which is why long-context NoPE needs attention-temperature scaling or hybrid layers.

The mechanism is easiest to read as counting. Row of the causal attention matrix has exactly unmasked entries, so a head whose logits are roughly constant across its window spreads of the probability mass onto each visible key. If the value vectors are dominated by one distinguished token, in practice the BOS token that every position can see, then the output norm at position is proportional to : a strictly decreasing, invertible function of absolute position that the next layer can consume as a positional feature. Kazemnejad et al. turn this observation into a theorem, and once an absolute code sits in the residual stream a later layer’s term can compute , recovering relative position too. Real NoPE models show the fingerprints of this construction: very heavy mass on the first token, first-layer heads with near-flat windows, and probes that decode absolute position from early activations.

Figure 1: The causal mask is the position signal. Counting unmasked entries per row gives under causal masking but a constant under bidirectional attention, so the same uniform head produces a strictly decreasing code in one case and a flat, uninformative constant in the other.

Mathematical Formulation:

Where:

• is the attention output at query position , and the attention weight it places on key position .
• are the query, key, and value vectors, containing no positional term at all under NoPE.
• index only the unmasked keys; the second line is the causal mask, which is the sole source of order sensitivity.
• is a constant logit, the degenerate case that makes attention uniform over the window; the third and fourth lines assume for and a nonzero anchor value at BOS.
• The fifth line inverts the code with , showing that absolute position is exactly recoverable from a single layer’s output norm.
• is a permutation matrix, the input sequence, an unmasked layer, and the lower-triangular mask; the last two lines state why encoders need explicit position encodings and causal decoders do not.

The same construction explains why NoPE is fragile at long context. Adjacent positions are separated by , a relative spacing of only , so the code that cleanly distinguishes position from position must resolve relative differences near around token 126 and near around token 1022. Under bf16 activations those differences sit at the edge of machine epsilon, and the second pressure is entropic: uniform attention over a growing window has entropy , so a head calibrated to be discriminative at 2K tokens is comparatively diffuse at 32K. Real models do not use the naive counting head alone, but both effects push in the same direction, which is why raw NoPE degrades past its training length while RoPE offers explicit rescaling recipes such as NTK interpolation and YaRN.

Figure 2: Two length pressures on an implicit code. The relative spacing between neighbouring positions shrinks like , crossing bf16 machine epsilon in the low hundreds of tokens, while the entropy ceiling grows like , so the same attention temperature that is sharp at 2K is diffuse at 32K.