DL0028 Sliding Window Attention

Sliding window attention restricts each token’s attention to a fixed-size local neighborhood instead of the whole sequence: token attends only to tokens within roughly positions of itself. This turns the dense attention matrix into a sparse band around the diagonal, cutting cost from to and making Transformers practical for very long documents, at the price of losing direct long-range links within a single layer.

(1) Local Window: Each token sees itself plus its nearest neighbors; everything else is masked out before the softmax.
(2) Linear Scaling in n: Cost per token is constant ( keys), so total compute and memory grow linearly with sequence length.
(3) Trade-off: Excellent for local patterns, but global context must be recovered by stacking layers, adding global tokens, or mixing with dilated windows.

Figure 1: Global attention fills the whole matrix; the sliding window keeps only a diagonal band of width .

Mathematical Formulation:

Where:

• is the window size (e.g., 512 in Longformer); tokens outside the window get scores before softmax.
• is the sequence length; with fixed, memory and compute scale linearly in .

How Global Context Survives: Stacking layers grows the effective receptive field to about tokens, similar to CNNs; Longformer additionally designates a few global tokens (e.g., [CLS], question tokens) that attend to and are attended by everything.

Figure 2: At , full attention needs ~268M score computations versus ~8.4M for a 512-wide window, a 32x saving that grows with .

Used In: Longformer, BigBird (window + global + random blocks), and many long-document and code models where full attention is infeasible.