DL0076 GQA: Grouped-Query Attention

Grouped-query attention splits the query heads of a layer into groups and gives each group a single shared key head and value head, so the number of distinct K/V projections drops from to while every query head keeps its own . It is the interpolation between two known extremes: is ordinary multi-head attention (MHA), and is multi-query attention (MQA). It exists because autoregressive decoding is memory-bandwidth bound, not compute bound: generating one token requires re-reading the entire KV cache from HBM, and that cache scales linearly with . MQA already fixed the bandwidth problem but lost quality and was unstable to train, so GQA was proposed to recover almost all of MQA’s decode speed at close to MHA quality by keeping a small number of K/V heads (typically 8). Nearly every major recent open-weight family ships with it, including Llama 3, where all sizes use 8 KV heads; the notable exception is DeepSeek-V2 and V3, which replace head sharing with multi-head latent attention.

(1) The Bottleneck It Targets: the KV cache, not the weights, is what grows with batch size and context, and decode re-reads all of it per token, so shrinking it by directly raises arithmetic intensity and tokens per second.
(2) A Dial Between Two Extremes: the lineage MHA → MQA → GQA is a single knob ; 8 KV heads recovers most of the memory win of 1 while avoiding the quality drop that MQA shows on summarization and long-input tasks.
(3) Cheap To Retrofit: an existing MHA checkpoint is converted by mean-pooling the K and V projections within each group and then uptraining on roughly 5% of the original pretraining tokens, so no full retrain is needed.
(4) Set To The Tensor-Parallel Degree: with equal to the number of shards, each GPU owns exactly one KV head and nothing is replicated; a smaller forces the same K/V to be duplicated across shards and gives back part of the saving.
(5) Almost No FLOP Change: the query projections and the product are untouched, so prefill latency barely moves; GQA is a memory and bandwidth optimization, not a compute one.

Figure 1: The only structural change is how many distinct K/V heads exist; query heads are never merged, so each still has its own projection and its own attention pattern over the shared keys.

Mathematical Formulation:

Where:

• is the number of query heads, the group size, and the resulting number of key/value heads; recovers MHA and recovers MQA.
• indexes query heads and is the KV head that head reads, so heads are assigned to groups by contiguous blocks.
• is the KV cache bytes per token, with layers, head dimension , bytes per element , and the factor 2 covering K and V; total cache is times batch times sequence length.
• is the set of query-head indices in group , and the last line is the mean-pooling initialization used when converting an MHA checkpoint; the same averaging is applied to .

The payoff is easiest to see with a concrete configuration. A Llama-3-70B-shaped model has , , and ; in FP16 that is 2.5 MiB per token of KV cache under MHA, 320 KiB under GQA with 8 KV heads, and 40 KiB under MQA. At a 32K context and batch size 1 those become 85.9 GB, 10.7 GB, and 1.34 GB. On a single H100 with 3.35 TB/s of HBM bandwidth, reading a 10.7 GB cache once costs roughly 3.2 ms per decoded token against roughly 26 ms for the 85.9 GB version; tensor parallelism divides both numbers, but the ratio is fixed by and does not improve with more hardware. That freed memory is also what lets a server hold many more concurrent sequences, which is usually a larger throughput win than the per-token latency itself.

Figure 2: FP16 KV cache for one 32K-token sequence on an 80-layer, 64-head, 128-dim model. Under MHA the cache alone would consume an entire 80 GB H100 before any weights are loaded.

Property	MHA	GQA (8 KV heads)	MQA
KV heads	64 (one per query head)	8 (group size 8)	1 (shared by all)
Cache per token (FP16)	2.5 MiB	320 KiB	40 KiB
Decode speed	Slowest, bandwidth bound on the cache	Close to MQA at long context	Fastest
Quality	Reference	Within noise of MHA on most benchmarks	Measurable drop, notably on long-input summarization
Tensor parallelism	Shards cleanly	Shards cleanly when equals the shard count	KV must be replicated on every shard
Training stability	Stable baseline	Stable; uptrainable from an MHA checkpoint	Reported instability at scale