DL0067 BERT VS GPT Pretraining

BERT is trained primarily with masked language modeling: selected input tokens are corrupted, and a bidirectional encoder predicts the original tokens from visible context on both sides. GPT uses causal language modeling: a decoder predicts each next token using only earlier tokens, enforced by a triangular attention mask. MLM supplies loss only at selected positions and creates a corruption mismatch between pre-training and ordinary inputs, whereas causal modeling supplies a target at nearly every position and matches left-to-right generation. Original BERT also used next sentence prediction, while standard GPT pre-training relies on the autoregressive token objective rather than NSP.

Figure 1: BERT reconstructs selected corrupted tokens from two-sided context, while GPT predicts the next token at every step from a left-only prefix.

(1) Context Visibility: BERT’s visible tokens attend bidirectionally; GPT position cannot attend to positions later than .
(2) Prediction Targets: BERT reconstructs a selected masked subset, while GPT shifts the sequence and predicts the next token at each usable position.
(3) Downstream Bias: BERT naturally supports full-context understanding, whereas GPT’s objective directly trains open-ended autoregressive generation; either family can be adapted beyond that bias.

Mathematical Formulation:

Where:

• and are the masked and causal language-modeling losses.
• is BERT’s selected target set and indexes one target token .
• denotes BERT’s visible corrupted context outside the selected targets, and is the reconstruction probability.
• indexes a GPT target position, is sequence length, and is the preceding token prefix.
• is the next-token probability and converts each probability into a log-likelihood term.

Figure 2: Attention visibility explains the objective difference: BERT uses a full visible-context matrix, while GPT uses a lower-triangular causal matrix.