DL0025 Attention Mechanism

Written by

Please explain the concept of “Attention Mechanism.”

Answer

The attention mechanism lets a model dynamically focus on the most relevant parts of the input when producing each output, instead of compressing the entire input into a single fixed-size context vector. For every output element, attention computes a weighted sum over input representations, where the weights (produced by a similarity score and a softmax) express how much each input element matters for that output. This solved the information bottleneck of early sequence-to-sequence models and is the core operation of Transformers.

(1) Query (Q): What the current output position is “looking for”: the element being processed.
(2) Key (K): An index of what each input position offers; queries are compared against keys to get similarity scores.
(3) Value (V): The actual content retrieved; scores are softmax-normalized into weights that blend the values into the output.

Bar chart of softmax attention weights over the five tokens I, love, machine, learning, models, with machine receiving 0.46 and learning 0.26 of the total weight.

Figure 1: Attention weights are a probability distribution over inputs: here “machine” and “learning” absorb 72% of the weight for a topic-focused query.

Mathematical Formulation (Scaled Dot-Product Attention):
$\mathrm{Attention}(Q, K, V) = \mathrm{softmax}\left(\frac{QK^{\top}}{\sqrt{d_k}}\right) V$

Where:

$Q, K, V$ are the query, key, and value matrices, linear projections of the token embeddings.
$QK^{\top}$ holds the raw dot-product similarity scores between every query-key pair.
$\sqrt{d_k}$ scales the scores (key dimension $d_k$ ) to keep softmax out of its saturated, tiny-gradient region; softmax normalizes each row into weights summing to 1.