DL0030 Positional Encoding

Written by

Explain “Positional Encoding” in Transformers. Why is it necessary?

Answer

Positional encoding injects token order into a Transformer’s input representations. Self-attention is permutation-invariant: shuffling the input tokens shuffles the outputs identically, so without positional information the model literally cannot tell “dog bites man” from “man bites dog”. Positional encodings add position-dependent vectors to the token embeddings before the Q, K, V projections, baking order awareness into every attention score.

(1) Fixed Sinusoidal: Parameter-free sine/cosine waves of geometrically spaced frequencies; encodes absolute position and extrapolates to unseen sequence lengths.
(2) Learned Embeddings: A trainable vector per position index, flexible and task-adaptive, but capped at the maximum training length.
(3) Relative Schemes: Encode distances between token pairs inside attention (RoPE, ALiBi) rather than absolute positions, the modern default in LLMs.

Heatmap of sinusoidal positional encoding values across 100 token positions and 64 embedding dimensions showing fast oscillations in low dimensions and slow stripes in high dimensions.

Figure 1: The sinusoidal PE matrix: low dimensions oscillate fast (fine position), high dimensions vary slowly (coarse position), giving every position a unique fingerprint.

Mathematical Formulation (Sinusoidal):
$PE_{(pos,\, 2i)} = \sin\left(\frac{pos}{10000^{2i / d_{model}}}\right)$
$PE_{(pos,\, 2i+1)} = \cos\left(\frac{pos}{10000^{2i / d_{model}}}\right)$

Where:

$pos$ is the token’s position in the sequence; $i$ indexes the embedding-dimension pair.
$d_{model}$ is the embedding dimension; wavelengths grow geometrically from $2\pi$ to $2\pi \cdot 10000$ across dimensions.

Why Multiple Frequencies: Each dimension pair is a wave of a different wavelength, so positions map to a unique multi-scale code; nearby positions also differ smoothly, giving the model an easy signal for relative offsets.

Three sine curves of positional encoding versus token position at different embedding dimensions with wavelengths ranging from about 6 to over 20000 positions.

Figure 2: Three PE dimensions as functions of position: the geometric frequency ladder lets the network read both fine and coarse order.

Where It Is Applied: $z_i = x_i + PE_i$ : add the encoding to each token embedding first, then compute $Q = ZW^Q$ , $K = ZW^K$ , $V = ZW^V$ , so position is present in every attention score from the start.