Category: Easy

DL0040 Attention Mask
What is the role of masking in attention?
Answer
Masking controls which positions attention is allowed to see. Before softmax, disallowed score entries are set to $-\infty$ (a very large negative number), so their attention weights become exactly zero. This one mechanism serves three jobs: preventing future leakage in autoregressive decoding, ignoring padding tokens in batched inputs, and enforcing task structure such as local neighborhoods or blocked spans.
(1) Causal Mask (Leakage Prevention): Blocks every position $j > i$ so token $i$ cannot read the future, which is mandatory for autoregressive training and decoding.
(2) Padding Mask: Blocks pad positions so they neither absorb probability mass nor inject meaningless context into real tokens.
(3) Structured Mask: Encodes task rules (local windows, graph neighborhoods, span blocking) by zeroing arbitrary score entries.
Mathematical Formulation:
$\mathrm{Attn}(Q, K, V, M) = \mathrm{softmax}\left(\frac{QK^\top}{\sqrt{d_k}} + M\right)V$
Where:
$M$ is the mask matrix: 0 for allowed positions, $-\infty$ for blocked ones; adding it before softmax drives blocked weights to zero.
$Q, K, V$ are the query/key/value matrices and $d_k$ the key dimension used for scaling.

Figure 1: Three mask patterns (rows = queries, columns = keys): padding blocks trailing columns, causal blocks the upper triangle, structured keeps only a local band.
Why $-\infty$ and Not Zero: Adding zero changes nothing, and deleting columns would break shapes; adding $-\infty$ makes $e^{-\infty} = 0$ inside softmax, so blocked positions get exactly zero weight while allowed positions renormalize cleanly among themselves.

Figure 2: Masking in the pipeline: scores are computed, the mask is added, softmax zeroes blocked entries, and only allowed values are mixed.
Combining Masks: In decoder training the causal and padding masks are summed (broadcast over queries) so a position is blocked if either rule forbids it; a fully masked row would produce NaNs, so real tokens always keep at least their own position unmasked.
Login to view more content
August 31, 2025
DL0038 Transformer Activation
Which activation functions do transformer models use?
Answer
Transformers use activations in two places. Inside the FFN, the hidden layer applies ReLU (original paper) or, in most modern models, GELU. BERT, GPT, and ViT all standardized on GELU, and newer LLMs adopt gated variants like SwiGLU. Inside attention, softmax normalizes the score matrix into attention weights. The GELU/ReLU choice controls gradient health; softmax controls score interpretability.
(1) GELU (FFN, modern default): Smooth, probabilistic gating: small negative inputs survive with tiny gradients, avoiding dead neurons and smoothing optimization.
(2) ReLU (FFN, original): $\max(0, x)$ , cheap and effective, but its hard zero can kill neurons permanently.
(3) Softmax (attention): Converts raw scores into a normalized distribution over keys, giving every query a convex combination of value vectors.

Figure 1: ReLU clamps negatives to a hard zero; GELU curves smoothly through them: small negative activations keep a non-zero gradient.
Mathematical Formulation:
$\mathrm{GELU}(x) = x \cdot \Phi(x) = x \cdot \frac{1}{2}\left[1 + \mathrm{erf}\left(\frac{x}{\sqrt{2}}\right)\right]$
$\mathrm{Softmax}(z_i) = \frac{e^{z_i}}{\sum_{j=1}^{K} e^{z_j}}$
Where:
$\Phi(x)$ is the standard Gaussian CDF: GELU gates the input by its own probability of being positive, a smooth stochastic-regularization view.
$z_i$ is one raw attention score and $K$ the number of scored keys; softmax outputs sum to 1.
Why GELU Won: Its smoothness yields non-zero gradients for negative inputs, reducing the “Dying ReLU” failure; the data-dependent gating acts like a soft, learned threshold. Empirically this means more stable training and better final loss in large language and vision models, which is why BERT/GPT-era models abandoned ReLU in the FFN.

Figure 2: The two activation sites in every block: softmax in attention, GELU/ReLU between the FFN’s two linear layers.
Beyond GELU: Recent LLMs (PaLM, LLaMA) replace the plain activation with gated linear units (SwiGLU/GEGLU), where one projection’s activation multiplies another linear path elementwise; this gating adds quality per parameter and has become the default in state-of-the-art FFN design.
Login to view more content
August 26, 2025

DL0036 Transformer Architecture II

What are the main differences between the encoder and decoder in a Transformer?

Answer

The encoder builds rich bidirectional representations of the source sequence: every token attends to every other token. The decoder is built for generation: its self-attention is causally masked so each position only sees the past, and it inserts an extra cross-attention sub-layer that reads the encoder’s output. Same building blocks, different wiring for two different jobs: understanding vs generating.

(1) Self-Attention Masking: Encoder self-attention is unmasked (full bidirectional context); decoder self-attention is masked so position $t$ attends only to $\leq t$ .
(2) Cross-Attention: Absent in the encoder; present in every decoder layer: queries from the decoder state, keys/values from the encoder output.
(3) Inputs & Role: Encoder consumes the source sequence once; decoder consumes the shifted-right target (teacher forcing) and produces next-token distributions.

Side-by-side encoder and decoder stacks separated by a dashed line, with the encoder's two sub-layers versus the decoder's three sub-layers and a cross-attention arrow bridging encoder output into the decoder.

Figure 1: Two stacks, three differences: the decoder adds a causal mask on self-attention and a cross-attention bridge to the encoder output.

Mathematical Formulation (decoder self-attention mask):
$\mathrm{Attn}(Q, K, V, M) = \mathrm{softmax}\left(\frac{QK^\top}{\sqrt{d_k}} + M\right)V$
$M_{ij} = 0 \;\text{if}\; j \leq i,\quad -\infty \;\text{if}\; j > i$

Where:

$M$ is the causal mask: $-\infty$ above the diagonal zeroes out future positions after softmax; the encoder simply omits $M$ .
$i, j$ index query and key positions; $j \leq i$ means “past or present only”.

Aspect	Encoder	Decoder
Self-attention	Unmasked: all positions	Masked: past positions only (causal)
Cross-attention	Not present	Present: attends to encoder outputs
Positional encoding	Added to source embeddings	Added to target embeddings (shifted right)
Input	Source sequence	Shifted target + encoder outputs
Function	Encode source into contextual representations	Generate target autoregressively with source context

Table 1: Encoder vs decoder at a glance. The decoder is a superset: same sub-layers plus masking and the cross-attention bridge.

Three mini diagrams showing query key value sources for encoder self-attention, masked decoder self-attention, and cross-attention with queries from the decoder and keys and values from the encoder output.

Figure 2: Q/K/V sources for the three attention types: only cross-attention mixes sequences: Q from the decoder, K/V from the encoder.

Why “Shifted Right”: During training the decoder receives the target sequence shifted one position right (prefixed with a start token), so the prediction at position $t$ is supervised against token $t$ while the input only ever reveals tokens before it. This is teacher forcing, and it keeps training fully parallel despite autoregressive inference.

August 17, 2025

DL0035 Transformer Architecture
Describe the original Transformer encoder–decoder architecture.
Answer
The original Transformer (Vaswani et al., 2017) is a sequence-to-sequence encoder–decoder built entirely from attention: no recurrence, no convolution. The encoder (6 stacked layers) reads the full source sequence and builds a contextual representation per token; the decoder (6 stacked layers) generates the target sequence one token at a time, attending both to its own past outputs and to the encoder’s representations.
(1) Encoder Layer: Multi-head self-attention (unmasked, so every token sees all positions) followed by a position-wise FFN, each wrapped in residual + LayerNorm.
(2) Decoder Layer: Masked multi-head self-attention (no peeking at future tokens), then cross-attention to the encoder output, then an FFN, again with residual + LayerNorm throughout.
(3) Input/Output: Token embeddings plus positional encodings feed both stacks; the decoder’s top passes through a linear + softmax head for next-token probabilities.

Figure 1: The original architecture: 6 encoder layers build source representations; 6 decoder layers generate autoregressively, bridged by cross-attention.
Mathematical Formulation (one sub-layer):
$z = \mathrm{LayerNorm}\big(x + \mathrm{Sublayer}(x)\big)$
Where:
$x$ is the sub-layer input ( $n \times d_{model}$ matrix of token vectors, $d_{model} = 512$ ).
$\mathrm{Sublayer}$ is multi-head self-attention, cross-attention, or the FFN; the residual add + LayerNorm pattern wraps every sub-layer in both stacks.

Figure 2: Inside one encoder layer: two sub-layers (MHA, FFN), each followed by add & norm; the decoder adds a masked self-attention and cross-attention stage.
Why the Design Works: Self-attention gives every token direct access to every other token (no information bottleneck through a fixed hidden state), residual connections + LayerNorm keep 12-deep training stable, and the masked decoder preserves causality so training can run in parallel via teacher forcing while inference stays autoregressive.
Login to view more content
August 14, 2025
DL0034 Layer Norm
What is layer normalization, and why is it used in Transformers?
Answer
Layer Normalization (LN) standardizes the features of each individual sample: for one token’s embedding vector, it computes the mean and variance across the feature dimension only, rescales to zero mean and unit variance, then applies a learnable scale and shift. Unlike BatchNorm, it never looks across the batch, which is exactly why Transformers, with variable-length sequences and small or on-the-fly batches, rely on it in every block.
(1) Normalization Within a Sample: Mean and variance come from the $d_{model}$ features of a single token: one set of statistics per token, not per batch.
(2) Batch-Size Independence: Behavior is identical at train and test time and for any batch size: no running statistics, no mismatch.
(3) Stabilizes Training: Keeps activations in a consistent range, preventing exploding/vanishing gradients and enabling deep stacks to converge faster.

Figure 1: Layer Normalization standardizes each sample (row) across its own feature dimensions; statistics never cross sample boundaries, so batch size is irrelevant.
Mathematical Formulation:
$\hat{x}_i = \frac{x_i - \mu}{\sqrt{\sigma^2 + \epsilon}} \cdot \gamma + \beta$
$\mu = \frac{1}{d}\sum_{i=1}^{d} x_i$
$\sigma^2 = \frac{1}{d}\sum_{i=1}^{d}(x_i - \mu)^2$
Where:
$x_i$ is one feature of a single token’s $d$ -dimensional vector; statistics are computed over $i = 1..d$ for that token alone.
$\epsilon$ is a small constant for numerical stability; $\gamma, \beta$ are learnable per-feature scale and shift that restore representational freedom.
Why Not BatchNorm: BatchNorm’s statistics mix information across samples, degrade with small or variable-size batches, behave differently at train vs inference, and pad tokens corrupt the per-feature means of variable-length sequences: all fatal for typical Transformer workloads.

Figure 2: Placement variants: original Post-LN (after the residual add) vs modern Pre-LN (inside the residual branch), which keeps a clean gradient highway.
Where It Sits: Every attention and FFN sub-layer is wrapped as $\mathrm{LN}(x + \mathrm{Sublayer}(x))$ (Post-LN) in the original paper; most modern LLMs use Pre-LN, normalizing the sub-layer input instead, which trains stably even without learning-rate warmup.
Login to view more content
August 14, 2025

DL0032 Transformer VS RNN

What makes Transformers more parallel-friendly than RNNs?

Answer

The fundamental difference is dependency structure: an RNN computes each hidden state from the previous one, $h_t = f(h_{t-1}, x_t)$ , so step $t$ cannot start before step $t-1$ finishes. A Transformer replaces recurrence with self-attention, which scores every pair of positions simultaneously, so all tokens are processed in one parallel pass. This turns sequential loops into dense matrix multiplications that saturate modern GPUs.

(1) No Temporal Dependency: Transformers process all input tokens at once; there is no hidden-state chain forcing order.
(2) Fully Parallelizable Attention: All $n^2$ attention scores are computed in a single matrix product $QK^\top$ , and the FFN applies to all positions simultaneously.
(3) Optimized for GPUs: Large GEMM kernels keep thousands of GPU cores busy, unlike the RNN’s long chain of small dependent steps.

Mathematical Formulation:
$h_t = f(h_{t-1},\, x_t)$
$S = QK^\top$

Where:

$h_t$ is the RNN hidden state at step $t$ ; it cannot be computed before $h_{t-1}$ exists.
$S$ is the full $n \times n$ attention score matrix, produced by one parallel GEMM from the query and key matrices $Q, K$ .

Side-by-side diagram of an unrolled RNN processing inputs one cell at a time through a hidden-state chain versus a Transformer block receiving all tokens at once in parallel.

Figure 1: RNN: a serial chain where each step waits for the previous hidden state; Transformer: the whole sequence enters the block simultaneously.

Training-Time Consequence: With a sequence of length $n$ , an RNN needs $n$ sequential steps no matter how much hardware you have: latency grows linearly with sequence length. A Transformer’s forward pass is a constant number of parallel matrix operations; the cost grows in FLOPs, not in wall-clock dependency depth.

Computation graph comparison showing the RNN as a vertical chain of dependent cell evaluations versus the Transformer as three wide parallel layers of matrix operations.

Figure 2: Dependency depth: RNN needs $n$ ordered steps; the Transformer needs only $O(1)$ sequential layers, each internally parallel.

Aspect	RNN	Transformer
Token dependency	Sequential: $h_t$ needs $h_{t-1}$	None: all tokens at once
Training steps for n tokens	$n$ ordered steps	One parallel pass
Core operation	Many small vector updates	Large batched GEMMs
Long-range signal path	Through $n$ hidden states (vanishing gradients)	One attention hop, $O(1)$ path length

Why It Matters: Parallelism is why Transformers can train on web-scale corpora (a workload that would take an RNN impractically long to finish), and why attention became the default sequence model in modern NLP.

August 10, 2025

DL0031 FFN in Transformer
What is the purpose of the feed-forward network inside each Transformer block?
Answer
The feed-forward network (FFN) inside each Transformer block processes every token independently after attention: it expands the token’s features into a higher-dimensional space, applies a non-linearity, and projects back to the model dimension. Attention mixes information across tokens; the FFN then deepens each token’s representation individually. It is where most of the block’s parameters and feature transformations actually happen.
(1) Non-Linear Transformation: Adds the only per-token non-linearity in the block, letting the model capture complex patterns that attention’s linear weighting cannot express.
(2) Token-Wise Processing: The same MLP is applied to each position separately: no cross-position mixing, so it parallelizes trivially over the sequence.
(3) Dimensional Expansion: The hidden layer typically expands $d_{model}$ by a factor of 4 (e.g., 512 → 2048), giving the network capacity to re-encode features before compressing them back.
Mathematical Formulation:
$\mathrm{FFN}(x) = \max(0,\; xW_1 + b_1)\, W_2 + b_2$
Where:
$x \in \mathbb{R}^{d_{model}}$ is one token’s vector after the attention sub-layer; the same function is applied to every position.
$W_1 \in \mathbb{R}^{d_{model} \times d_{ff}}$ , $W_2 \in \mathbb{R}^{d_{ff} \times d_{model}}$ are trainable weights; typically $d_{ff} = 4\, d_{model}$ .
$\max(0, \cdot)$ is the ReLU activation of the original paper; modern models swap in GELU or gated variants like SwiGLU.

Figure 1: $d \to 4d \to d$ : expand, activate, compress. The same two-layer MLP for every token position.
Why the Expansion Matters: Projecting up to $d_{ff}$ gives the network a wide workspace to recombine and re-weight the features attention produced; the second matrix then distills the result back to $d_{model}$ . Roughly two-thirds of a vanilla Transformer’s parameters live in these FFN matrices.
Complement to Attention: Multi-head attention is a weighted average of value vectors, a linear operation per head. Without the FFN, stacking attention layers would compose mostly linear maps; the FFN’s expansion + non-linearity supplies the representational depth that makes deep stacks worthwhile.
Login to view more content
August 8, 2025
DL0030 Positional Encoding
Explain “Positional Encoding” in Transformers. Why is it necessary?
Answer
Positional encoding injects token order into a Transformer’s input representations. Self-attention is permutation-invariant: shuffling the input tokens shuffles the outputs identically, so without positional information the model literally cannot tell “dog bites man” from “man bites dog”. Positional encodings add position-dependent vectors to the token embeddings before the Q, K, V projections, baking order awareness into every attention score.
(1) Fixed Sinusoidal: Parameter-free sine/cosine waves of geometrically spaced frequencies; encodes absolute position and extrapolates to unseen sequence lengths.
(2) Learned Embeddings: A trainable vector per position index, flexible and task-adaptive, but capped at the maximum training length.
(3) Relative Schemes: Encode distances between token pairs inside attention (RoPE, ALiBi) rather than absolute positions, the modern default in LLMs.

Figure 1: The sinusoidal PE matrix: low dimensions oscillate fast (fine position), high dimensions vary slowly (coarse position), giving every position a unique fingerprint.
Mathematical Formulation (Sinusoidal):
$PE_{(pos,\, 2i)} = \sin\left(\frac{pos}{10000^{2i / d_{model}}}\right)$
$PE_{(pos,\, 2i+1)} = \cos\left(\frac{pos}{10000^{2i / d_{model}}}\right)$
Where:
$pos$ is the token’s position in the sequence; $i$ indexes the embedding-dimension pair.
$d_{model}$ is the embedding dimension; wavelengths grow geometrically from $2\pi$ to $2\pi \cdot 10000$ across dimensions.
Why Multiple Frequencies: Each dimension pair is a wave of a different wavelength, so positions map to a unique multi-scale code; nearby positions also differ smoothly, giving the model an easy signal for relative offsets.

Figure 2: Three PE dimensions as functions of position: the geometric frequency ladder lets the network read both fine and coarse order.
Where It Is Applied: $z_i = x_i + PE_i$ : add the encoding to each token embedding first, then compute $Q = ZW^Q$ , $K = ZW^K$ , $V = ZW^V$ , so position is present in every attention score from the start.
Login to view more content
August 7, 2025
DL0027 Multi-Head Attention
How does multi-head attention work in transformer architectures?
Answer
Multi-head attention runs several attention operations in parallel on different learned projections of the same input: Q, K, V are projected into $h$ lower-dimensional subspaces (one per head), each head performs scaled dot-product attention independently, and the $h$ outputs are concatenated and linearly re-projected by $W^O$ . Each head can specialize in a different relationship (syntax, local neighborhoods, long-range links) so the combined representation is richer than any single attention map.
(1) Split: Project $d_{model}$ -dim Q, K, V into $h$ heads of dimension $d_k = d_{model} / h$ each.
(2) Parallel Attention: Every head computes its own softmax-weighted sum; all heads run simultaneously, so total compute is comparable to one full-size head.
(3) Merge: Concatenate the head outputs and apply output projection $W^O$ back to $d_{model}$ .

Figure 1: The MHA pipeline: split → parallel attention → concat → project; each head attends in its own subspace.
Mathematical Formulation:
$\mathrm{MultiHead}(Q, K, V) = \mathrm{Concat}(\mathrm{head}_1, \dots, \mathrm{head}_h)\, W^O$
$\mathrm{head}_i = \mathrm{Attention}(Q W_i^Q,\; K W_i^K,\; V W_i^V)$
Where:
$W_i^Q, W_i^K, W_i^V \in \mathbb{R}^{d_{model} \times d_k}$ are the per-head projection matrices; $h$ is the number of heads.
$d_k = d_{model} / h$ is each head’s subspace dimension (e.g., 512-dim model with 8 heads → $d_k = 64$ ).
$W^O \in \mathbb{R}^{h d_k \times d_{model}}$ mixes the concatenated heads back to the model dimension.
Why Multiple Heads Help: A single softmax distribution must average every relationship into one map; separate heads let the model attend to different positions and relation types simultaneously. Empirically, heads specialize into recognizable patterns.

Figure 2: Same input, four different attentions: heads specialize on position bands, special tokens, and semantic pairs that one head alone could not capture.
Login to view more content
July 31, 2025

DL0026 Self-Attention vs Cross-Attention

What distinguishes self-attention from cross-attention in transformer models?

Answer

The distinction is where Q, K, and V come from. In self-attention, all three are projections of the same sequence, so every token weighs every other token within its own sequence, modeling internal dependencies. In cross-attention, the queries come from one sequence (e.g., the decoder state) while the keys and values come from a different sequence (e.g., the encoder output), letting one representation selectively read from another. Both use the identical scaled dot-product computation.

(1) Input Scope: Self-attention: Q, K, V from one sequence; cross-attention: Q from the target sequence, K/V from the source sequence.
(2) Architectural Role: Self-attention appears in encoder and decoder blocks; cross-attention is the bridge that injects encoder context into each decoder step.
(3) Score Matrix Shape: Self-attention produces an $n \times n$ map; cross-attention produces $n_{dec} \times n_{enc}$ , rectangular when lengths differ.

Side-by-side attention heatmaps: self-attention among the tokens The cat sat on the mat, and cross-attention from decoder queries to the encoder sequence The animal rested on a rug.

Figure 1: Same mechanism, different sources: self-attention relates tokens within one sentence; cross-attention aligns decoder tokens to a separate encoder sequence.

Mathematical Formulation:
$\mathrm{Attention}(Q, K, V) = \mathrm{softmax}\left(\frac{QK^{\top}}{\sqrt{d_k}}\right) V$
$Q = X_{q} W^Q, \quad K = X_{kv} W^K, \quad V = X_{kv} W^V$

Where:

$X_q$ is the query-source sequence; $X_{kv}$ is the key/value-source sequence; self-attention is the special case $X_q = X_{kv}$ .
$W^Q, W^K, W^V$ are learnable projection matrices; $d_k$ is the key dimension used for scaling.

Side-by-Side Comparison:

Aspect	Self-Attention	Cross-Attention
Q source	Same sequence	Target sequence (e.g., decoder)
K, V source	Same sequence	Different sequence (e.g., encoder output)
Models	Intra-sequence dependencies	Inter-sequence alignment
Score matrix	$n \times n$ (square)	$n_{dec} \times n_{enc}$ (rectangular)
Typical location	Encoder & decoder blocks	Decoder blocks (bridging to encoder)

Bottom line: self-attention answers “which of my own tokens matter to me?”; cross-attention answers “which tokens of the other sequence should I read now?”

July 31, 2025