DL0063 Transformer Variable Length Sequences

A Transformer can process different sequence lengths because self-attention is defined over whatever number of tokens is supplied, up to the model’s context limit. For efficient batching, implementations usually pad sequences to a common length or group examples of similar lengths, then apply a padding mask so valid queries cannot attend to padded keys and values. Decoder-style models also add a causal mask that blocks future positions, while positional encodings identify token order within each sequence. Padded query outputs and losses must be ignored, and computation still scales with the padded batch length rather than only the number of valid tokens.

(1) Batch Construction: Sequences are dynamically padded to the longest item in a batch, bucketed by length, or packed by specialized kernels to reduce wasted work.
(2) Attention Masking: A key-padding mask removes padded keys and values from attention; causal models combine it with a triangular future mask.
(3) Length Boundary: Variable length does not mean unlimited length: positional support, memory, and the configured context window impose a maximum.

Figure 1: Unequal token sequences become a rectangular batch; masking separates valid context, padding, and future positions.

Mathematical Formulation:

Where:

• , , and are query, key, and value matrices, and is the key width used for score scaling.
• assigns to padded key positions so their softmax probabilities become zero.
• assigns when key index is later than query index ; encoder-only models normally omit this mask.
• denotes one combined mask entry and is either for an allowed connection or for a blocked connection.
• is batch size and is padded sequence length, giving a dense token tensor of shape .

Figure 2: A practical variable-length pipeline: tokenize, bucket or pad, build masks, run attention, then ignore padded outputs or continue autoregressive generation.