DL0043 KV Cache

The KV cache stores the key and value vectors of all previously generated tokens so that autoregressive decoding never recomputes them. At each step the model computes Q, K, V only for the new token, appends K and V to the cache, and attends over the cached history, turning each decoding step from an full recompute into an lookup, at the price of growing memory.

(1) What Is Cached: Per layer, the projected keys and values of every past token; they never change once computed, so recomputing them is pure waste.
(2) Why It Helps: With causal masking, token attends to exactly ; caching reduces each step to one new query against a read-only cache.
(3) The Cost: Memory grows linearly ( values per layer for K and V, across layers) and becomes the inference bottleneck for long contexts and large batches.

Mathematical Formulation:

Where:

• is the query of the current token ; are the cached keys and values for tokens .
• is the per-head key dimension, the model width, and the number of layers; the factor 2 counts K and V buffers separately.

Figure 1: Without cache every step recomputes the full ; with cache only the new token’s query is computed while cached K/V (purple) supply the history.

Practical Note: During training the full sequence is processed in parallel so the cache does not apply; it is an inference-only optimization, and its size is why long-context serving is memory-bound rather than FLOP-bound.