DL0029 Dilated Attention

Dilated attention sparsifies self-attention by letting each query attend only to every -th key position (a strided subset of the sequence) instead of all keys or a contiguous window. Borrowed from dilated convolutions in CNNs, the dilation rate controls the stride: attention keeps a long global reach (the sampled keys span the whole sequence) while computing only about of the score matrix. The trade-off is granularity: nearby fine detail is skipped within each head.

(1) Strided Sampling: Query attends to keys where is a multiple of ; recovers full attention.
(2) Global but Sparse: Unlike sliding windows, coverage spans the entire sequence: long-range links survive, sampled coarsely.
(3) Cost Reduction: Each row computes roughly scores, so compute and memory drop from toward .

Figure 1: Dilation : each query attends every 3rd key: full-span coverage with a third of the computations.

Mathematical Formulation:

Where:

• is the dilation rate, the stride between attended key positions.
• are the dilated subsets of keys and values (about rows per query).
• is the key dimension, used for the usual softmax scaling.

Coverage Through Stacking: Layers with increasing dilation rates (e.g., 1, 2, 4, 8) progressively widen and interleave coverage, so deeper layers see the full sequence even though each layer is sparse: the same trick as WaveNet’s exponentially dilated convolutions.

Figure 2: Gridding artifacts: repeating one fixed dilation leaves some positions permanently unattended. Mixing dilation rates closes the gaps.