DL0077 LLM Prediction Objective

A large language model (LLM) is a neural network, in practice a decoder-only Transformer with billions of parameters, trained on trillions of tokens of text to model the probability distribution over token sequences. Its prediction objective is next-token prediction: factor the sequence probability with the chain rule and maximize the log-likelihood of each observed token given its left context, which is identical to minimizing cross-entropy over the vocabulary at every position. The training signal is self-supervised, since the label at position is simply the token at position , so no human annotation is required and one sequence of length yields supervised examples in a single forward pass, or a full when a leading BOS token supplies the context for the first real token. Everything an LLM appears to know (syntax, factual associations, translation, arithmetic, code structure) is a byproduct of driving that one loss down, because accurately predicting the next token over a broad corpus requires modeling the processes that generated the text. Base pretraining optimizes likelihood only; instruction following and refusal behavior come later, from supervised fine-tuning and preference optimization, which change the objective rather than the architecture.

(1) What Makes It “Large”: parameter count in the billions, a training corpus in the trillions of tokens, and a compute budget allocated according to scaling laws, so capability improvements are largely predictable from loss curves rather than architectural novelty.
(2) The Objective Is Autoregressive: the chain-rule factorization plus a causal mask means position may only attend to positions through , which is what makes single-pass training and left-to-right generation consistent with each other.
(3) Loss Is Cross-Entropy Over the Vocabulary: the final layer projects the hidden state to logits, softmax turns them into a distribution, and the loss is the negative log-probability of the single correct token.
(4) Perplexity Is the Reported Metric: is the effective branching factor, so a mean loss of nats means the model is about as uncertain as choosing uniformly among 10 tokens; perplexity depends on the tokenizer and is not comparable across different vocabularies.
(5) Likelihood Is Not Helpfulness: maximizing the likelihood of scraped text also rewards reproducing its errors and its style, and teacher forcing never trains the model to recover from its own mistakes, which is the origin of exposure bias.

Figure 1: Illustrative output of one position: a full distribution over the whole vocabulary. Because the observed continuation is “Paris” with probability 0.83, this position contributes nats to the loss.

Two properties of this objective explain most of how LLMs are trained and evaluated. First, supervision is dense: the causal mask lets one forward pass score every shifted position simultaneously, so a 2048-token document produces 2047 gradient contributions (2048 with a BOS prepended), which is why raw web text is enough to train a model with hundreds of billions of parameters. Second, the loss is a strictly proper scoring rule on the token distribution, so the gradient pushes probability mass toward the observed token and away from every competitor at once; the model is never told which wrong token was “almost right”. That makes the loss an excellent optimization target and a poor proxy for downstream quality, which is why teams track perplexity for training health but use task benchmarks and human preference for release decisions.

Mathematical Formulation:

Where:

• is the token sequence produced by the tokenizer, and is the left context of position ; the first line is the chain-rule factorization that defines an autoregressive model.
• The term assumes a leading BOS token as context; without it the empty-context term is dropped and the sum runs over positions with the mean taken over terms.
• is the final-layer hidden state at position , and with bias is the unembedding head, frequently weight-tied to the input embedding.
• is the vocabulary size (roughly 128k for Llama 3, about 50k for GPT-2) and the model width; collects all parameters.
• is the mean cross-entropy in nats per token, equal to the negative log-likelihood because the target is a one-hot distribution; dividing by converts it to bits per token.
• ranges from 1 (a perfect model) up to for a uniform untrained model, so is the loss value at initialization and a useful sanity check.

Figure 2: Teacher forcing on one short sequence: the target row is the input row shifted by one, so a 6-token sequence yields 5 scored positions, every position gets its own cross-entropy term, and the sequence loss is their mean, here 0.97 nats, giving a perplexity of about 2.6.