DL0040 Attention Mask

Masking controls which positions attention is allowed to see. Before softmax, disallowed score entries are set to (a very large negative number), so their attention weights become exactly zero. This one mechanism serves three jobs: preventing future leakage in autoregressive decoding, ignoring padding tokens in batched inputs, and enforcing task structure such as local neighborhoods or blocked spans.

(1) Causal Mask (Leakage Prevention): Blocks every position so token cannot read the future, which is mandatory for autoregressive training and decoding.
(2) Padding Mask: Blocks pad positions so they neither absorb probability mass nor inject meaningless context into real tokens.
(3) Structured Mask: Encodes task rules (local windows, graph neighborhoods, span blocking) by zeroing arbitrary score entries.

Mathematical Formulation:

Where:

• is the mask matrix: 0 for allowed positions, for blocked ones; adding it before softmax drives blocked weights to zero.
• are the query/key/value matrices and the key dimension used for scaling.

Figure 1: Three mask patterns (rows = queries, columns = keys): padding blocks trailing columns, causal blocks the upper triangle, structured keeps only a local band.

Why and Not Zero: Adding zero changes nothing, and deleting columns would break shapes; adding makes inside softmax, so blocked positions get exactly zero weight while allowed positions renormalize cleanly among themselves.

Figure 2: Masking in the pipeline: scores are computed, the mask is added, softmax zeroes blocked entries, and only allowed values are mixed.

Combining Masks: In decoder training the causal and padding masks are summed (broadcast over queries) so a position is blocked if either rule forbids it; a fully masked row would produce NaNs, so real tokens always keep at least their own position unmasked.