DL0078 LLM Training Stages

A modern LLM is built in a sequence of stages that share one architecture but differ in data, objective, and scale: pretraining on a web-scale corpus with next-token prediction, an increasingly explicit mid-training phase that reweights the data mix and extends the context window, supervised fine-tuning (SFT) on curated prompt-response pairs, and preference optimization that turns human or programmatic judgments into a training signal (reward model plus PPO, DPO, or reinforcement learning with verifiable rewards). The split is not cosmetic: pretraining consumes well over 90% of the total tokens and FLOPs and is where almost all knowledge and reasoning capacity is acquired, while post-training uses a tiny fraction of the compute to select and expose behaviors the base model already has. Each stage produces a named artifact that the next stage consumes, so the usual lineage is base model → long-context base → instruct model → aligned model. Practical pipelines also interleave evaluation, safety filtering, distillation, and quantization, but those are packaging steps rather than new learning objectives.

(1) Pretraining: self-supervised next-token loss on trillions of filtered web, code, and book tokens, run once at enormous cost; this stage fixes the tokenizer, the parameter count, and the knowledge cutoff.
(2) Mid-Training: continued pretraining on a higher-quality mix with upsampled math, code, and long documents, plus context-length extension and learning-rate annealing, using roughly 1-10% of the pretraining tokens.
(3) Supervised Fine-Tuning: the same cross-entropy loss but computed only on response tokens with the prompt masked out, teaching format, instruction following, and tool-call syntax from to curated examples.
(4) Preference Optimization: optimizes a reward (learned from pairwise comparisons, or computed by a verifier) under a KL penalty toward the SFT reference policy, which is what shifts a model from plausible to preferred.
(5) Compute Asymmetry: post-training is cheap, so it is where iteration happens; anything that requires new knowledge or a longer context has to go back to a pretraining-style stage.

Figure 1: The four canonical stages with their input data, objective, and output artifact. Only the data distribution and loss mask change between stages 1 to 3; stage 4 replaces likelihood with a reward under a KL constraint.

Mathematical Formulation:

Where:

• is the preceding context and the target token, so pretraining averages this loss over every position of every document.
• is the prompt, the response, and the set of assistant token positions; positions outside are masked, which is the only structural difference between SFT and pretraining.
• and are the preferred and rejected responses for the same prompt, is the reward model, and is the logistic function, giving the Bradley-Terry pairwise objective.
• is the policy being trained and the frozen SFT checkpoint; the KL term is what keeps generations fluent instead of collapsing onto reward-model artifacts.
• sets the strength of that anchor: small permits reward hacking, large leaves the model barely changed from SFT.

Figure 2: Illustrative token budget for a frontier-scale run. Pretraining and mid-training together see about 99.97% of all tokens, so post-training cannot add missing knowledge, only elicit and shape what is already in the weights.