DL0041 Hierarchical Attention

Written by

Could you explain the concept of hierarchical attention in transformer architectures?

Answer

Hierarchical attention applies self-attention at multiple levels of granularity instead of one flat pass over all tokens: first local attention within segments (words inside a sentence, frames inside a shot), then global attention across the aggregated segment representations (sentences inside a document). This mirrors the natural structure of long inputs, cuts the quadratic cost dramatically, and yields interpretable focus at each level.

(1) Local Level (Fine-Grained): Each segment runs its own self-attention over its tokens, producing one segment embedding; cost grows with segment length, not document length.
(2) Global Level (Coarse-Grained): The segment embeddings attend over each other, producing a document-level representation.
(3) Efficiency Gain: A document of $n$ tokens split into $s$ segments of $m$ tokens costs $O(s m^2 + s^2)$ in attention entries instead of $O(n^2)$ , a large saving when $m \ll n$ .

Two-level hierarchy diagram: word tokens attend inside sentence segments, sentence embeddings then attend globally to form a document embedding.

Figure 1: Two attention levels: local attention inside each segment, then global attention over segment embeddings to form the document representation.

Mathematical Formulation (cost for n = s × m tokens):
$\text{Flat:}\quad \mathrm{Cost} \propto n^2$
$\text{Hierarchical:}\quad \mathrm{Cost} \propto \underbrace{s\, m^2}_{\text{local}} + \underbrace{s^2}_{\text{global}} \ll n^2$

Where:

$n$ is the total token count, split into $s$ segments of $m$ tokens each ( $n = s \cdot m$ ).
The local term runs $s$ independent $m \times m$ attentions; the global term runs one $s \times s$ attention over segment embeddings.

Example (Document Classification): With $n = 4096$ tokens as $s = 64$ sentences of $m = 64$ words, flat attention scores $16.8\text{M}$ pairs, while hierarchical attention scores only $64 \cdot 64^2 + 64^2 \approx 0.27\text{M}$ , roughly 60x fewer pairs.