DL0027 Multi-Head Attention

Multi-head attention runs several attention operations in parallel on different learned projections of the same input: Q, K, V are projected into lower-dimensional subspaces (one per head), each head performs scaled dot-product attention independently, and the outputs are concatenated and linearly re-projected by . Each head can specialize in a different relationship (syntax, local neighborhoods, long-range links) so the combined representation is richer than any single attention map.

(1) Split: Project -dim Q, K, V into heads of dimension each.
(2) Parallel Attention: Every head computes its own softmax-weighted sum; all heads run simultaneously, so total compute is comparable to one full-size head.
(3) Merge: Concatenate the head outputs and apply output projection back to .

Figure 1: The MHA pipeline: split → parallel attention → concat → project; each head attends in its own subspace.

Mathematical Formulation:

Where:

• are the per-head projection matrices; is the number of heads.
• is each head’s subspace dimension (e.g., 512-dim model with 8 heads → ).
• mixes the concatenated heads back to the model dimension.

Why Multiple Heads Help: A single softmax distribution must average every relationship into one map; separate heads let the model attend to different positions and relation types simultaneously. Empirically, heads specialize into recognizable patterns.

Figure 2: Same input, four different attentions: heads specialize on position bands, special tokens, and semantic pairs that one head alone could not capture.