DL0044 Multi-Query Attention

Multi-Query Attention (MQA) is a variant of multi-head attention where all query heads share a single key and value projection instead of each head owning its own K/V. The KV cache therefore stores one K/V pair per token rather than one per head, shrinking cache memory and bandwidth by a factor of , which directly speeds up autoregressive decoding, usually with only a minor quality cost.

(1) Structure: Queries are still projected into distinct heads, but all of them attend against one shared K and one shared V.
(2) Cache Savings: The per-token KV cache drops from to , x smaller.
(3) Trade-off: Slightly less expressive than full MHA (one key/value “view” of the sequence), which GQA later interpolates by sharing K/V across small groups of heads.

Mathematical Formulation:

Where:

• is the query of head ; are the single key/value projections used by all heads.
• is the head dimension and the number of query heads.

Figure 1: MHA gives every head its own K and V; MQA keeps multiple query heads but shares one K and one V across all of them.

Why Decoding Gets Faster: Autoregressive inference is memory-bandwidth bound: every generated token must read the entire KV cache. An x smaller cache means x less data movement per step, which is why MQA (and its successor GQA, used in LLaMA-2/3) is standard in serving-oriented LLMs.