DL0037 Transformer Architecture III

Written by

Why do Transformers use a dot product, rather than addition, to compute attention scores?

Answer

The dot product is a natural similarity measure: $q \cdot k$ is large exactly when query and key point in the same direction, so it directly expresses “how relevant is this key to my query”. It is also a single fused matrix multiplication ( $QK^\top$ scores all pairs at once), which is far faster on modern hardware than additive attention’s per-pair MLP. Scaling by $1/\sqrt{d_k}$ keeps it numerically stable in high dimensions.

(1) Geometric Meaning: $q \cdot k = \|q\|\,\|k\|\cos\theta$ : aligned vectors score high, orthogonal vectors score zero, opposed vectors score negative.
(2) Hardware Efficiency: All scores materialize as one GEMM $QK^\top$ ; additive attention needs an MLP evaluated for every (query, key) pair.
(3) Probabilistic Readout: Softmax over dot-product scores yields interpretable attention weights: a distribution over which keys matter.

2D vector diagram showing three vectors from the origin with a positive dot product between two aligned vectors and a negative dot product between two opposed vectors.

Figure 1: Dot product as similarity: aligned vectors score positive, opposed vectors score negative; softmax turns scores into attention weights.

Mathematical Formulation:
$\alpha_i = \frac{e^{q \cdot k_i / \sqrt{d_k}}}{\sum_{j=1}^{n} e^{q \cdot k_j / \sqrt{d_k}}}$

Where:

$\alpha_i$ is the attention weight the query assigns to key $k_i$ ; weights sum to 1 over the $n$ keys.
$q \cdot k_i$ is the dot-product similarity; $\sqrt{d_k}$ scaling prevents softmax saturation for large key dimension $d_k$ .

Why Not Addition: Adding features component-wise, $q + k$ , says nothing about alignment; two very different pairs can sum to the same vector. To become a usable score, the sum must pass through an extra learnable MLP ( $v^\top \tanh(W_q q + W_k k)$ ), which is slower per pair and still lacks the dot product’s direct geometric interpretation.

Comparison flowchart of dot-product attention scoring all pairs in one matrix multiply versus additive attention pushing each query-key pair through a small neural network scorer.

Figure 2: Dot-product scoring = one batched GEMM; additive scoring = a per-pair MLP, richer but much slower at scale.

The One Caveat: For large $d_k$ , raw dot products grow in variance proportional to $d_k$ , pushing softmax into saturated, gradient-poor regions, which is why the Transformer uses the scaled dot product rather than the plain one.

Transformer

DL0037 Transformer Architecture III

Comments

Leave a Reply Cancel reply

More posts

MSD0064 Notification Ranking System

MSD0063 Image Copyright Detection

MSD0062 Ads Ranking System

MSD0061 Nearby Place Recommendation