Tag: System

The design is a CI pipeline for a stochastic system: a curated golden suite mined from real production failures, a grader ladder (deterministic checks where possible, an LLM judge calibrated against human labels where necessary, human audits on top), and a release gate that compares the candidate against the current version on quality metrics plus guardrails (safety, latency, cost). GitHub runs more than 4,000 such offline tests in CI before any Copilot model change; Anthropic’s guidance is to start with 20 to 50 tasks drawn from real failures and grow from there. The pipeline ends in a canary and A/B test, and production monitoring feeds new failures back into the suite, so the eval set is a living artifact.

(1) Cases From Reality: the golden suite is mined from production logs, bug reports, and support tickets, not invented at a desk; 20-50 real tasks beat 500 synthetic ones.
(2) Grader Ladder: deterministic checks first, an LLM judge (calibrated to human labels, pairwise where possible) second, human audits as the calibration anchor.
(3) Capability vs Regression Suites: capability evals start at a low pass rate and measure progress; regression evals sit near 100% and block backsliding; saturated capability cases graduate into the regression suite.
(4) Release Gate With Guardrails: the candidate must beat or tie the incumbent on quality with statistical significance, and must not breach guardrails on safety, latency, and cost.
(5) Flywheel: production failures are mined, de-identified, and converted into new eval cases every release cycle.

Figure 1: The pre-release path: golden suite, grader ladder, gate, then canary. The loop back from production logs is what keeps the suite honest as user behavior shifts.

Clarify Before Designing:
(1) Product Surface: single-turn answers, multi-turn chat, or an agent with tool calls; the grader design differs sharply.
(2) What Changes per Release: model version, prompts, retrieval data, tools; each change type needs its own regression coverage.
(3) Failure Cost: what is the worst plausible wrong answer (refund mistakes, medical tone, leaked PII) and who is accountable for it?
(4) Human Label Budget: how many expert labels per cycle can we afford, since they anchor the judge’s calibration?
(5) Release Cadence: weekly prompt tweaks and quarterly model upgrades imply different gate depths.

The design is a model ladder behind a per-sentence router: cheap deterministic rules and dictionaries catch spelling and punctuation on-device, an edit-tagging Transformer (one parallel pass over the tokens emitting KEEP, DELETE, INSERT, REPLACE tags) handles most grammatical error correction in the cloud, a seq2seq model takes the sentences that need reordering, and an LLM serves style and clarity rewrites only on demand. The two pivotal decisions are routing each sentence to the cheapest model that can fix it (latency is a product feature), and tuning every error type to a precision-first operating point because false corrections erode trust faster than misses.

(1) Error Taxonomy Drives Architecture: mechanical errors (spelling, punctuation, agreement) and open-ended style rewrites are different problems and deserve different models.
(2) Model Ladder With Routing: rules → edit-tagger → seq2seq GEC → LLM, cheapest first; the router reads error type, confidence, and latency budget.
(3) Precision-First Operating Points: per-error-type thresholds tuned on F_0.5, since a wrong correction damages trust more than a miss.
(4) Latency Engineering: score only the current sentence, cache unchanged text, and keep the inline path under 100 ms end to end.
(5) Personalization & Dialect: locale (US/UK), register (casual vs formal), and per-user dismiss history modulate which suggestions fire.

Figure 1: The inline path: each finished sentence is routed to the cheapest sufficient model, and a merger dedupes overlapping suggestions before rendering underlines. Unchanged text is served from cache.

Clarify Before Designing:
(1) Scope: which languages and locales, and is style or clarity rewriting in scope or only mechanical correctness?
(2) Latency Budget: how many milliseconds for inline suggestions, and is there an async path for long-form checks?
(3) Deployment: what runs on-device versus in the cloud, given browser-extension and mobile constraints?
(4) Privacy: may user text leave the device, and are there regulated contexts (legal, medical) to special-case?
(5) Personalization: can we learn from per-user accept and dismiss behavior, and where may that profile live?

The system is a real-time classification pipeline at the mail gateway. It begins with data collection and preprocessing (features from text and metadata) and trains a supervised model on user-labeled history. Deployment is a low-latency prediction service that scores every incoming email and applies a tuned threshold. Because spammers constantly adapt, the system lives or dies by its monitoring and continuous-training loop that fights concept drift.

(1) Objectives & Metrics: Precision first: a false positive (legitimate mail in spam) is far costlier than a false negative; keep recall high as the secondary goal and track F1 / PR-AUC.
(2) Data & Features: User-labeled history plus external corpora; features span email text (subject, body), metadata (sender, domain reputation, recipients), and behavior (links, attachments, frequency).
(3) Two-Stage Cascade: A fast traditional model filters the obvious ~80%, then a Transformer deep-analyzes the ambiguous ~20%: accuracy where it matters, speed everywhere else.
(4) Deployment: A low-latency prediction API at the gateway: receive → extract features → score → threshold → route to spam folder or inbox.
(5) Maintenance: Monitor precision/recall daily, harvest user feedback (‘Mark as Spam’ / ‘Not Spam’) as fresh labels, and retrain continuously against concept drift.

Figure 1: The real-time inference path: every incoming email is scored by the model (e.g., 0.95), and a tuned threshold (e.g., 0.8) routes it (spam folder or inbox) inside the latency budget of the mail gateway.

Clarify Before Designing:
(1) Scale: email volume per day and peak QPS at the gateway?
(2) Latency Budget: how many milliseconds may scoring add before the email is delivered?
(3) Cost Asymmetry: confirmed: is a false positive (lost legitimate mail) really the catastrophic error here?
(4) Labels: how much user-labeled spam/not-spam data exists, and how fresh is it?
(5) Adversaries: how quickly do spam tactics mutate in this ecosystem: days, weeks?