DL0094 Evaluating an LLM

There is no single number, so I treat evaluation as a stack of layers chosen to match how the model will actually be used: intrinsic likelihood (perplexity or bits per byte) at the bottom, static capability benchmarks above it, automated graders such as unit tests and LLM-as-judge, human pairwise preference, and finally an online A/B test on the real product metric. Each layer up the stack is closer to user value and more expensive, so cheap layers act as fast regression gates and expensive layers make ship decisions. The practical work is defining the task metric first, then building a private evaluation set drawn from production traffic, because public leaderboards measure a distribution that is rarely yours. Three things decide whether an evaluation is worth anything: contamination control, judge calibration against human labels, and error bars. Most reported “improvements” of one or two points on a 1000-item benchmark are inside the noise band and would not survive a paired significance test.

(1) Start From The Deployment Task: a RAG assistant is scored on groundedness, citation precision, and refusal correctness, a coding model on pass@k against hidden tests, an agent on end-to-end task completion; a generic leaderboard score answers none of these.
(2) Intrinsic Versus Extrinsic: perplexity is cheap and useful for pretraining and quantization regressions, but it is not comparable across tokenizers and correlates weakly with instruction-following quality.
(3) Static Benchmarks Decay: MMLU-Pro, GPQA, MATH, HumanEval, and SWE-bench Verified are reproducible and cheap, but they saturate and they leak into pretraining corpora, so contamination checks (n-gram overlap, held-out variants, release-date filtering) are mandatory.
(4) Automated Graders Scale, With Bias: the loop is sample responses → grade with a rubric or unit tests → aggregate with intervals; a judge model shows position, verbosity, and self-preference bias, so swap the answer order and calibrate agreement with humans before trusting it.
(5) Human Preference Is The Open-Ended Gold Standard: pairwise votes fitted with a Bradley-Terry (Elo) model give a single latent quality score, at the cost of slow turnaround and annotator disagreement.
(6) Report Uncertainty, Safety, And Cost: every score needs a confidence interval and a paired test against the baseline, plus a separate axis for jailbreak resistance, toxicity, PII leakage, and the p95 latency and cost per request that decide whether the win is affordable.

Figure 1: The evaluation stack. Cheap layers at the bottom run on every commit as regression gates; the expensive layers at the top are the only ones that measure user value, so a release decision needs evidence from both ends.

Two failure modes dominate real evaluation work. The first is contamination: a model that has memorized a benchmark scores high without generalizing, which you detect by rewriting items into paraphrases or numeric variants and watching the score collapse, or by using benchmarks whose items post-date the training cutoff. The second is judge miscalibration: an LLM judge that prefers longer, more confidently worded answers will happily rank a verbose model above a correct one, so I always measure judge-human agreement (Cohen’s kappa above roughly 0.6 before the judge is allowed to gate a release) and control for response length. On top of that, generation is stochastic, so a benchmark run at temperature 0.7 has run-to-run variance of its own; fix seeds and decoding parameters, or report the mean over several runs. Finally, offline wins must be confirmed online, because the production metric that matters (resolution rate, edit distance to the accepted answer, escalation rate) frequently moves in the opposite direction from a benchmark score.

Mathematical Formulation:

Where:

• is perplexity over a held-out sequence , with indexing tokens; it depends on the tokenizer, so bits per byte is the cross-model comparable form.
• is the number of samples drawn per coding problem, the number that pass the hidden tests, and the budget of attempts scored; this is the unbiased pass@k estimator, averaged over problems.
• is the modelled probability that response beats , and are the fitted Bradley-Terry ratings on the Elo scale, where a 100-point gap implies about a 64% win rate.
• is the observed accuracy on independent items and its standard error; the 95% interval is , which for and is roughly points.
• Required condition: items must be independent and not contaminated; when two models are scored on the same items, use a paired test (McNemar or a paired bootstrap) rather than comparing two independent intervals.

Figure 2: Accuracy with 95% intervals on a 1000-item benchmark. The 2.3-point gap between the first two models sits well inside the noise band, so an unpaired claim of improvement is unsupported; only the third model separates cleanly.