DL0026 Self-Attention vs Cross-Attention

The distinction is where Q, K, and V come from. In self-attention, all three are projections of the same sequence, so every token weighs every other token within its own sequence, modeling internal dependencies. In cross-attention, the queries come from one sequence (e.g., the decoder state) while the keys and values come from a different sequence (e.g., the encoder output), letting one representation selectively read from another. Both use the identical scaled dot-product computation.

(1) Input Scope: Self-attention: Q, K, V from one sequence; cross-attention: Q from the target sequence, K/V from the source sequence.
(2) Architectural Role: Self-attention appears in encoder and decoder blocks; cross-attention is the bridge that injects encoder context into each decoder step.
(3) Score Matrix Shape: Self-attention produces an map; cross-attention produces , rectangular when lengths differ.

Figure 1: Same mechanism, different sources: self-attention relates tokens within one sentence; cross-attention aligns decoder tokens to a separate encoder sequence.

Mathematical Formulation:

Where:

• is the query-source sequence; is the key/value-source sequence; self-attention is the special case .
• are learnable projection matrices; is the key dimension used for scaling.

Side-by-Side Comparison:

Aspect	Self-Attention	Cross-Attention
Q source	Same sequence	Target sequence (e.g., decoder)
K, V source	Same sequence	Different sequence (e.g., encoder output)
Models	Intra-sequence dependencies	Inter-sequence alignment
Score matrix	(square)	(rectangular)
Typical location	Encoder & decoder blocks	Decoder blocks (bridging to encoder)

Bottom line: self-attention answers “which of my own tokens matter to me?”; cross-attention answers “which tokens of the other sequence should I read now?”