DL0079 Scaling Laws Size vs Data

Neural scaling laws say that test loss falls as a power law in both parameter count and training tokens , plus an irreducible floor, so the two are not independent knobs: a fixed compute budget forces a trade between them. Kaplan et al. (2020) fit the loss surface and concluded that most extra compute should go into parameters, which is why GPT-3 used 175B parameters on only about 300B tokens (roughly 1.7 tokens per parameter). Hoffmann et al. (2022, “Chinchilla”) re-ran the sweep with properly decayed learning-rate schedules over more than 400 runs and found the compute-optimal frontier scales both quantities at roughly the same rate, and , giving the well-known heuristic of about 20 tokens per parameter. Under that law a 70B model trained on 1.4T tokens beat a 280B model trained on 300B tokens at equal compute. The practical answer to the interview question is therefore: data requirements grow roughly linearly with model size along the compute-optimal frontier, and any deviation from that ratio should be justified by a constraint the loss law does not model, such as a finite corpus or the cost of serving.

(1) Joint Power Law: loss decomposes into an entropy floor plus one shrinking term per resource, , so starving either resource leaves its term dominant and no amount of the other resource can fix it.
(2) Compute Couples The Two: with FLOPs (2 for the forward pass and 4 for the backward pass per parameter per token), choosing fixes , which is why the question is always “what split”, never “how big”.
(3) IsoFLOP Minima Define The Frontier: sweeping at fixed traces a shallow U-shaped curve; the locus of minima across budgets is the compute-optimal frontier, and its slope in log-log space is the exponent you actually care about.
(4) Kaplan Versus Chinchilla: the earlier exponents () came from a fixed-length cosine schedule truncated early and from parameter counts that excluded embeddings, both of which penalize the long-token runs and bias the fit toward oversized models.
(5) Optimal Does Not Mean Correct: the law optimizes training loss per training FLOP; it says nothing about inference cost, so serving-heavy models are deliberately overtrained far past 20 tokens per parameter.

The empirical procedure behind these numbers is worth stating precisely, because interviewers probe it. You pick a grid of compute budgets, and for each budget you train several models of different widths and depths, always to the token count that exhausts exactly that budget, with the learning rate decayed to its minimum at the end of each run. Plotting final loss against for one budget yields an IsoFLOP curve whose minimum is flat enough that a factor of two in parameter count often costs under 1% in loss, which is exactly the slack engineers exploit when memory or latency constraints bite.

Figure 1: Illustrative IsoFLOP curves from with . Each budget’s minimum sits at , moving from 0.29B parameters / 5.8B tokens at FLOPs to 9.1B parameters / 183B tokens at FLOPs, and the basins are shallow near the optimum.

Mathematical Formulation:

Where:

• is expected test loss in nats per token, and is the irreducible term: the entropy of the data plus whatever the architecture can never represent, so no scaling drives loss to zero.
• is the number of trainable parameters (embeddings included, which is where the Kaplan fit differed) and is the number of training tokens.
• are fitted scale constants and are the power-law exponents; Chinchilla’s fit put both near , meaning the two resources have comparable marginal value.
• is training compute in FLOPs; the factor 6 comes from roughly 2 FLOPs per parameter per token forward and 4 backward, and it ignores attention’s quadratic term, which is minor while dominates sequence length.
• are the frontier exponents obtained by minimizing subject to the compute constraint; always holds, so if parameters take a larger share of compute growth, tokens must take a smaller one.
• The ratio is constant only when ; the “20 tokens per parameter” rule is that special case, and it drifts slowly with budget under asymmetric fits.

Figure 2: Illustrative comparison of the two prescriptions, normalized to agree at FLOPs. Because the exponents differ ( versus ), the gap compounds: at FLOPs one recipe asks for 91B parameters on 1.8T tokens and the other for 1.3T parameters on 130B tokens, about a disagreement in both directions.

Modern frontier practice has moved past the compute-optimal point on purpose. If a model will serve billions of tokens, total lifetime FLOPs are dominated by inference, and a smaller model trained on far more data is cheaper end to end even though its training run is nominally suboptimal. Meta’s Llama 3 8B was trained on roughly 15T tokens, near 1,875 tokens per parameter, almost two orders of magnitude past the Chinchilla ratio, and the authors report the loss was still improving log-linearly at that point. The scaling law is still doing the work here; the objective being minimized has simply changed from training loss per training FLOP to quality per unit of deployment cost.