Your team ships an LLM-powered product: a customer-support copilot built on a foundation model API, similar in spirit to how GitHub evaluates models for Copilot before each release. The model version, prompts, retrieval index, and tools all change every few weeks, and every change can silently alter answer quality.
Design an evaluation pipeline that runs before each release. What do you test, how do you score open-ended outputs, and how do you decide whether the new version is safe to ship?
The Problem: the system under test is stochastic, so a green build means nothing without a repeatable quality signal. Design the gate between “we changed the prompt” and “customers see it.”
Answer
The design is a CI pipeline for a stochastic system: a curated golden suite mined from real production failures, a grader ladder (deterministic checks where possible, an LLM judge calibrated against human labels where necessary, human audits on top), and a release gate that compares the candidate against the current version on quality metrics plus guardrails (safety, latency, cost). GitHub runs more than 4,000 such offline tests in CI before any Copilot model change; Anthropic’s guidance is to start with 20 to 50 tasks drawn from real failures and grow from there. The pipeline ends in a canary and A/B test, and production monitoring feeds new failures back into the suite, so the eval set is a living artifact.
(1) Cases From Reality: the golden suite is mined from production logs, bug reports, and support tickets, not invented at a desk; 20-50 real tasks beat 500 synthetic ones.
(2) Grader Ladder: deterministic checks first, an LLM judge (calibrated to human labels, pairwise where possible) second, human audits as the calibration anchor.
(3) Capability vs Regression Suites: capability evals start at a low pass rate and measure progress; regression evals sit near 100% and block backsliding; saturated capability cases graduate into the regression suite.
(4) Release Gate With Guardrails: the candidate must beat or tie the incumbent on quality with statistical significance, and must not breach guardrails on safety, latency, and cost.
(5) Flywheel: production failures are mined, de-identified, and converted into new eval cases every release cycle.
Figure 1: The pre-release path: golden suite, grader ladder, gate, then canary. The loop back from production logs is what keeps the suite honest as user behavior shifts.
Clarify Before Designing:
(1) Product Surface: single-turn answers, multi-turn chat, or an agent with tool calls; the grader design differs sharply.
(2) What Changes per Release: model version, prompts, retrieval data, tools; each change type needs its own regression coverage.
(3) Failure Cost: what is the worst plausible wrong answer (refund mistakes, medical tone, leaked PII) and who is accountable for it?
(4) Human Label Budget: how many expert labels per cycle can we afford, since they anchor the judge’s calibration?
(5) Release Cadence: weekly prompt tweaks and quarterly model upgrades imply different gate depths.
Leave a Reply