DL0052 Rotary Positional Embedding

Rotary Positional Embedding (RoPE) encodes position by rotating each pair of query/key dimensions by an angle proportional to the token’s position, instead of adding a position vector. Because a rotation at position composed with one at position depends only on the difference, the attention dot product between tokens automatically reflects their relative distance, with no learned parameters and good length generalization. RoPE is the default in GPT-NeoX, LLaMA, PaLM, Qwen, and most modern LLMs.

(1) Geometric Encoding: Each 2D slice of a query or key vector is rotated by , where is the token position and the per-dimension frequency.
(2) Relative From Absolute: The dot product of two rotated vectors depends on only: absolute rotations yield relative-position attention.
(3) Parameter-Free and Efficient: No embedding table, no attention bias, just a fixed trigonometric transform applied to Q and K.

Mathematical Formulation:

Where:

• is the token’s absolute position in the sequence; are one 2D pair of the query/key vector.
• is the rotation frequency of the -th dimension pair, geometrically decreasing across pairs so low dimensions capture short-range and high dimensions long-range offsets.

Figure 1: Similarity vs relative distance: RoPE decays smoothly with distance (a built-in recency prior), while sinusoidal absolute PE stays roughly flat.

Why It Won: RoPE is shift-invariant (the same rotation rule works at any position), adds zero parameters or memory, and, unlike learned absolute embeddings, degrades gracefully beyond the training length, which is why context-extension methods (NTK scaling, YaRN) all build on rescaling its frequencies.