DL0035 Transformer Architecture

The original Transformer (Vaswani et al., 2017) is a sequence-to-sequence encoder–decoder built entirely from attention: no recurrence, no convolution. The encoder (6 stacked layers) reads the full source sequence and builds a contextual representation per token; the decoder (6 stacked layers) generates the target sequence one token at a time, attending both to its own past outputs and to the encoder’s representations.

(1) Encoder Layer: Multi-head self-attention (unmasked, so every token sees all positions) followed by a position-wise FFN, each wrapped in residual + LayerNorm.
(2) Decoder Layer: Masked multi-head self-attention (no peeking at future tokens), then cross-attention to the encoder output, then an FFN, again with residual + LayerNorm throughout.
(3) Input/Output: Token embeddings plus positional encodings feed both stacks; the decoder’s top passes through a linear + softmax head for next-token probabilities.

Figure 1: The original architecture: 6 encoder layers build source representations; 6 decoder layers generate autoregressively, bridged by cross-attention.

Mathematical Formulation (one sub-layer):

Where:

• is the sub-layer input ( matrix of token vectors, ).
• is multi-head self-attention, cross-attention, or the FFN; the residual add + LayerNorm pattern wraps every sub-layer in both stacks.

Figure 2: Inside one encoder layer: two sub-layers (MHA, FFN), each followed by add & norm; the decoder adds a masked self-attention and cross-attention stage.

Why the Design Works: Self-attention gives every token direct access to every other token (no information bottleneck through a fixed hidden state), residual connections + LayerNorm keep 12-deep training stable, and the masked decoder preserves causality so training can run in parallel via teacher forcing while inference stays autoregressive.