The standard Transformer’s self-attention mechanism has a computational and memory complexity of 
, where 
is the sequence length. For long document classification (e.g., thousands of tokens), this quadratic scaling becomes prohibitive.
Describe one or more attention modifications you would design or choose to enable efficient and effective long document classification.
The Problem: full self-attention costs 
in compute and memory. Redesign attention so a several-thousand-token document stays affordable without giving up classification accuracy.
Answer
To handle long documents, the quadratic complexity of full self-attention () must be reduced. The two primary directions are Sparse Attention (Longformer, BigBird), which constrains each token to a relevant subset (a local window plus a few global tokens), and Hierarchical Attention (HATN), which applies attention within segments first and then across segment representations. Both preserve classification quality while cutting cost from toward linear; linearized and recurrent variants (Performer, Linformer, Transformer-XL) attack the same wall from other angles.
(1) Sparse Attention (Mechanism): Replace the full attention matrix with a sparse design: local window attention for nearby context plus global tokens (e.g., [CLS]) that exchange information with all tokens (Longformer, BigBird).
(2) Hierarchical Attention (Structure): Split the document into segments; apply token-level attention within each segment, then a document-level attention across segment representations (HATN, LNLF-BERT).
(3) Linearized / Recurrent Variants: Kernel approximations (Performer, Linformer) reach true linear complexity; Transformer-XL / Compressive Transformer carry a recurrent memory across segments.
Figure 1: Full attention (left) fills the whole 
matrix; sliding-window attention (right) keeps only a diagonal band: each token attends to its neighbors, and cost drops from to 
.
Clarify Before Designing:
(1) Document Length: typical and maximum token count: 2k, 8k, 32k?
(2) Starting Point: adapt a pretrained BERT-style encoder, or train from scratch?
(3) Binding Constraint: memory at training, or latency at inference?
(4) Interaction Pattern: does classification need global token interactions (a [CLS] readout), and how long-range are the dependencies?
(5) Budget For Change: may we modify the architecture, or only fine-tune an off-the-shelf long model?
Leave a Reply