Tag: Transformer

The distinction is where Q, K, and V come from. In self-attention, all three are projections of the same sequence, so every token weighs every other token within its own sequence, modeling internal dependencies. In cross-attention, the queries come from one sequence (e.g., the decoder state) while the keys and values come from a different sequence (e.g., the encoder output), letting one representation selectively read from another. Both use the identical scaled dot-product computation.

(1) Input Scope: Self-attention: Q, K, V from one sequence; cross-attention: Q from the target sequence, K/V from the source sequence.
(2) Architectural Role: Self-attention appears in encoder and decoder blocks; cross-attention is the bridge that injects encoder context into each decoder step.
(3) Score Matrix Shape: Self-attention produces an map; cross-attention produces , rectangular when lengths differ.

Figure 1: Same mechanism, different sources: self-attention relates tokens within one sentence; cross-attention aligns decoder tokens to a separate encoder sequence.

Mathematical Formulation:

Where:

• is the query-source sequence; is the key/value-source sequence; self-attention is the special case .
• are learnable projection matrices; is the key dimension used for scaling.

Side-by-Side Comparison:

Aspect	Self-Attention	Cross-Attention
Q source	Same sequence	Target sequence (e.g., decoder)
K, V source	Same sequence	Different sequence (e.g., encoder output)
Models	Intra-sequence dependencies	Inter-sequence alignment
Score matrix	(square)	(rectangular)
Typical location	Encoder & decoder blocks	Decoder blocks (bridging to encoder)

Bottom line: self-attention answers “which of my own tokens matter to me?”; cross-attention answers “which tokens of the other sequence should I read now?”

The attention mechanism lets a model dynamically focus on the most relevant parts of the input when producing each output, instead of compressing the entire input into a single fixed-size context vector. For every output element, attention computes a weighted sum over input representations, where the weights (produced by a similarity score and a softmax) express how much each input element matters for that output. This solved the information bottleneck of early sequence-to-sequence models and is the core operation of Transformers.

(1) Query (Q): What the current output position is “looking for”: the element being processed.
(2) Key (K): An index of what each input position offers; queries are compared against keys to get similarity scores.
(3) Value (V): The actual content retrieved; scores are softmax-normalized into weights that blend the values into the output.

Figure 1: Attention weights are a probability distribution over inputs: here “machine” and “learning” absorb 72% of the weight for a topic-focused query.

Mathematical Formulation (Scaled Dot-Product Attention):

Where:

• are the query, key, and value matrices, linear projections of the token embeddings.
• holds the raw dot-product similarity scores between every query-key pair.
• scales the scores (key dimension ) to keep softmax out of its saturated, tiny-gradient region; softmax normalizes each row into weights summing to 1.