Category: NLP/LLM

The design is 3D parallelism mapped onto the bandwidth hierarchy, wrapped in an elastic supervisor that treats hardware failure as a routine event. Tensor parallelism stays inside a node where NVLink is cheap, pipeline parallelism crosses nodes because only stage activations travel, and data parallelism with sharded optimizer state sits outermost where the gradient all-reduce can be overlapped with the backward pass. Fault tolerance is asynchronous distributed checkpointing at a mathematically chosen interval plus a hot-spare pool, so a dead GPU costs minutes rather than the run. The single number the whole system is managed against is MFU (model FLOPs utilization), because at this scale every lost percentage point is real money, and every design choice below is justified by its effect on that number.

(1) Bandwidth-Matched Parallelism: tensor parallel degree capped at the intra-node GPU count, pipeline parallel across nodes, data parallel outermost; context parallelism added only when sequences get long.
(2) Sharded State And Selective Recompute: optimizer states and gradients sharded across the data-parallel group, with selective activation recomputation trading roughly 8% extra FLOPs for the memory that makes the fit possible.
(3) Async Checkpointing At The Optimal Interval: copy state to host memory in seconds, flush to storage in the background, and set the interval from measured cluster MTBF instead of habit.
(4) Elastic Recovery With Hot Spares: a watchdog detects hangs and stragglers, a spare node replaces the failed one in place, and training resumes from the last checkpoint including data-loader position.
(5) Topology-Aware Gang Scheduling: the job is placed inside one network island as an all-or-nothing gang, after a preflight burn-in that rejects slow links and weak GPUs before the run starts.
(6) MFU As The Run’s SLO: per-rank step-time histograms, profiler traces, and a loss budget that attributes every missing point of peak FLOPs to a named cause.

Figure 1: One step is load → forward → backward → reduce → update → stage checkpoint. The lower loop is what makes a month-long run survivable: detect, replace, restore.

Clarify Before Designing:
(1) Hardware And Interconnect: how many GPUs per node, what intra-node link (NVLink or PCIe), and what inter-node fabric bandwidth per GPU?
(2) Token Budget And Deadline: how many training tokens, and is the constraint calendar time, total GPU-hours, or dollars?
(3) Sequence Length: is context 8k or 128k, because long context changes activation memory and adds a whole parallelism axis?
(4) Precision Policy: is FP8 or BF16 allowed for GEMMs, and what loss-curve deviation from a reference run is acceptable?
(5) Storage And Checkpoint SLA: what aggregate write bandwidth exists, and how much lost work per failure is tolerable?
(6) Cluster Ownership: dedicated reservation or shared preemptible pool, and how large a hot-spare pool can be held idle?

The design is a staged pipeline with a federated summarization stack, the approach Zoom publishes for AI Companion: streaming ASR turns audio into words, speaker diarization attributes them to people, a post-processing stage restores punctuation and domain terms, and the summarization stage drafts on a small in-house LLM, scores the draft with a learned quality model (Zoom’s Z-Scorer), and escalates to a stronger external LLM only when the score is low. The two pivotal decisions are separating “who spoke” from “what was said” (diarization plus ASR reconciled with the meeting roster, not one giant model asked to guess names) and never paying frontier-model prices for meetings a small model summarizes fine (Zoom reports comparable quality at roughly 6% of GPT-4 cost).

(1) ASR Frontend: streaming encoder-decoder (Whisper-class) with custom vocabulary biasing for product and people names; about 95% word accuracy on real meetings.
(2) Speaker Diarization: embedding-based turn detection (pyannote-style) reconciled with the participant roster and per-channel audio when available, so “Speaker 2” becomes “Ada”.
(3) Structured Summarization: chained tasks with schema-checked JSON output: chapter segmentation, recap, decisions, action items with owners.
(4) Federated Quality Control: small model drafts, a learned scorer predicts summary quality, low scores escalate to stronger LLMs, and a committee-of-LLMs cross-check suppresses hallucinated action items.
(5) Privacy by Inheritance: summaries inherit the meeting’s access control and retention; a self-hosted-only model path serves regulated customers.

Figure 1: The pipeline: hear the words, attribute them, then summarize with schema-checked structure; the scorer, not blind trust, decides when a stronger model is worth paying for.

Clarify Before Designing:
(1) Live versus Post: in-meeting “catch me up” queries (seconds, incremental) or post-meeting delivery only (minutes, batch)?
(2) Language Scope: English only, or 30-plus languages via translation assists or multilingual models?
(3) Identity Source: is a participant roster, per-channel audio, or voice enrollment available, or must speakers stay anonymous labels?
(4) Worst Error: which failure is most expensive here: hallucinated action items, missed decisions, or wrong attribution?
(5) Scale and Budget: meetings per day, average length, and the cost ceiling per meeting-minute?

The design is a trained per-request router: a tiny scorer reads the request and predicts the probability that the strong model’s answer beats the weak one’s, then sends the request to the cheapest model expected to clear the quality bar. This is routing, not cascading, so each request normally pays for exactly one model. The two pivotal decisions are to train the router on human preference data (which model’s answer people actually preferred, as in Chatbot Arena battles) rather than hand-built rules, and to expose an explicit cost-quality operating threshold as a product knob, with an escalation fallback so a routing mistake costs extra compute, not a bad answer.

(1) Difficulty Signals: query embedding and length, task type, tool needs, and conversation state; the router is BERT-class or smaller, ~1/1000 the cost of a strong call.
(2) Preference-Trained Router: learn P(strong model wins | query) from human preference battles, augmented with LLM-judged comparisons; route strong when P passes the threshold.
(3) Explicit Operating Point: the threshold is chosen on the measured cost-quality curve and recalibrated as the traffic mix drifts.
(4) Guarded Fallback: if the small model or a verifier flags low confidence, escalate to the large model; routing errors become cost, not bad answers.
(5) Model Registry & Shadowing: candidates live behind a versioned registry; new models are shadow-scored on live traffic before the router is retrained or recalibrated.

Figure 1: The routing path: one cheap score per request, one paid model call, and an escalation hatch when the cheap path doubts itself.

Clarify Before Designing:
(1) Candidate Set: two models or many; closed APIs, self-hosted weights, or both; is the real split fast-versus-reasoning rather than small-versus-large?
(2) Decision Granularity: per request, per message in a conversation, or pinned per session for style consistency?
(3) Quality Bar: which evaluation defines “good enough” (human preference, task success), and how much loss versus the always-strong policy is tolerable?
(4) Router Budget: how much latency and cost may the router itself add per request?
(5) Traffic Shape: what fraction of traffic is trivial chat versus hard reasoning today, and how fast does that mix drift?

The design is a single massively multilingual encoder-decoder model trained on mined bitext plus back-translated monolingual text, with a target-language token steering generation (the NLLB recipe), behind a dual-track serving layer: a fast NMT track for short, latency-critical, terminology-sensitive text, and a fine-tuned translation LLM for long, nuanced, document-level input, which is exactly the split Google shipped in its Cloud Translation API (2024) and DeepL shipped as its next-gen model. The two pivotal decisions are the multilingual pooling (high-resource pairs teach low-resource ones through a shared representation, enabling zero-shot pairs) and the honest admission that one model family does not win everywhere: NMT still wins on latency, cost, and determinism, the LLM wins on context and fluency.

(1) One Multilingual Model: shared encoder-decoder with a target-language token; transfer from high-resource pairs gives zero-shot coverage of unseen pairs instead of 10,000 separate models.
(2) Data Flywheel: mine bitext from web crawls with multilingual sentence embeddings (LASER), then back-translate monolingual text to synthesize pairs for low-resource languages.
(3) Dual-Track Serving: NMT for short/interactive/terminology-critical requests, translation LLM for long-form and idiomatic text; a router picks per request.
(4) Quality Measurement: automatic metrics (COMET, BLEU) for regression, human MQM error marking for truth, A/B tests for product impact, all reported per language pair.
(5) Low-Resource Strategy: zero-shot transfer first, then few-example adaptive conditioning (Google’s Adaptive Translation shows as few as 5, and up to 30k, example pairs can steer output at inference time).

Figure 1: Dual-track serving over one multilingual training base: the router sends short, latency-critical text to NMT and long, context-heavy text to the translation LLM; terminology constraints apply on both tracks.

Clarify Before Designing:
(1) Scope: text only, or also speech and image translation; and which language pairs at launch?
(2) Latency Budget: interactive typing (sub-300 ms) versus document translation (seconds) imply different model families.
(3) Quality Bar: gist-level understanding for users, or publication-grade output with human post-editing?
(4) Terminology: must product names, legal terms, or brand glossaries be honored exactly?
(5) Customization: do enterprise customers need domain adaptation (medical, legal) with their own example pairs?

The design is a CI pipeline for a stochastic system: a curated golden suite mined from real production failures, a grader ladder (deterministic checks where possible, an LLM judge calibrated against human labels where necessary, human audits on top), and a release gate that compares the candidate against the current version on quality metrics plus guardrails (safety, latency, cost). GitHub runs more than 4,000 such offline tests in CI before any Copilot model change; Anthropic’s guidance is to start with 20 to 50 tasks drawn from real failures and grow from there. The pipeline ends in a canary and A/B test, and production monitoring feeds new failures back into the suite, so the eval set is a living artifact.

(1) Cases From Reality: the golden suite is mined from production logs, bug reports, and support tickets, not invented at a desk; 20-50 real tasks beat 500 synthetic ones.
(2) Grader Ladder: deterministic checks first, an LLM judge (calibrated to human labels, pairwise where possible) second, human audits as the calibration anchor.
(3) Capability vs Regression Suites: capability evals start at a low pass rate and measure progress; regression evals sit near 100% and block backsliding; saturated capability cases graduate into the regression suite.
(4) Release Gate With Guardrails: the candidate must beat or tie the incumbent on quality with statistical significance, and must not breach guardrails on safety, latency, and cost.
(5) Flywheel: production failures are mined, de-identified, and converted into new eval cases every release cycle.

Figure 1: The pre-release path: golden suite, grader ladder, gate, then canary. The loop back from production logs is what keeps the suite honest as user behavior shifts.

Clarify Before Designing:
(1) Product Surface: single-turn answers, multi-turn chat, or an agent with tool calls; the grader design differs sharply.
(2) What Changes per Release: model version, prompts, retrieval data, tools; each change type needs its own regression coverage.
(3) Failure Cost: what is the worst plausible wrong answer (refund mistakes, medical tone, leaked PII) and who is accountable for it?
(4) Human Label Budget: how many expert labels per cycle can we afford, since they anchor the judge’s calibration?
(5) Release Cadence: weekly prompt tweaks and quarterly model upgrades imply different gate depths.

The design is a model ladder behind a per-sentence router: cheap deterministic rules and dictionaries catch spelling and punctuation on-device, an edit-tagging Transformer (one parallel pass over the tokens emitting KEEP, DELETE, INSERT, REPLACE tags) handles most grammatical error correction in the cloud, a seq2seq model takes the sentences that need reordering, and an LLM serves style and clarity rewrites only on demand. The two pivotal decisions are routing each sentence to the cheapest model that can fix it (latency is a product feature), and tuning every error type to a precision-first operating point because false corrections erode trust faster than misses.

(1) Error Taxonomy Drives Architecture: mechanical errors (spelling, punctuation, agreement) and open-ended style rewrites are different problems and deserve different models.
(2) Model Ladder With Routing: rules → edit-tagger → seq2seq GEC → LLM, cheapest first; the router reads error type, confidence, and latency budget.
(3) Precision-First Operating Points: per-error-type thresholds tuned on F_0.5, since a wrong correction damages trust more than a miss.
(4) Latency Engineering: score only the current sentence, cache unchanged text, and keep the inline path under 100 ms end to end.
(5) Personalization & Dialect: locale (US/UK), register (casual vs formal), and per-user dismiss history modulate which suggestions fire.

Figure 1: The inline path: each finished sentence is routed to the cheapest sufficient model, and a merger dedupes overlapping suggestions before rendering underlines. Unchanged text is served from cache.

Clarify Before Designing:
(1) Scope: which languages and locales, and is style or clarity rewriting in scope or only mechanical correctness?
(2) Latency Budget: how many milliseconds for inline suggestions, and is there an async path for long-form checks?
(3) Deployment: what runs on-device versus in the cloud, given browser-extension and mobile constraints?
(4) Privacy: may user text leave the device, and are there regulated contexts (legal, medical) to special-case?
(5) Personalization: can we learn from per-user accept and dismiss behavior, and where may that profile live?

The system is a real-time classification pipeline at the mail gateway. It begins with data collection and preprocessing (features from text and metadata) and trains a supervised model on user-labeled history. Deployment is a low-latency prediction service that scores every incoming email and applies a tuned threshold. Because spammers constantly adapt, the system lives or dies by its monitoring and continuous-training loop that fights concept drift.

(1) Objectives & Metrics: Precision first: a false positive (legitimate mail in spam) is far costlier than a false negative; keep recall high as the secondary goal and track F1 / PR-AUC.
(2) Data & Features: User-labeled history plus external corpora; features span email text (subject, body), metadata (sender, domain reputation, recipients), and behavior (links, attachments, frequency).
(3) Two-Stage Cascade: A fast traditional model filters the obvious ~80%, then a Transformer deep-analyzes the ambiguous ~20%: accuracy where it matters, speed everywhere else.
(4) Deployment: A low-latency prediction API at the gateway: receive → extract features → score → threshold → route to spam folder or inbox.
(5) Maintenance: Monitor precision/recall daily, harvest user feedback (‘Mark as Spam’ / ‘Not Spam’) as fresh labels, and retrain continuously against concept drift.

Figure 1: The real-time inference path: every incoming email is scored by the model (e.g., 0.95), and a tuned threshold (e.g., 0.8) routes it (spam folder or inbox) inside the latency budget of the mail gateway.

Clarify Before Designing:
(1) Scale: email volume per day and peak QPS at the gateway?
(2) Latency Budget: how many milliseconds may scoring add before the email is delivered?
(3) Cost Asymmetry: confirmed: is a false positive (lost legitimate mail) really the catastrophic error here?
(4) Labels: how much user-labeled spam/not-spam data exists, and how fresh is it?
(5) Adversaries: how quickly do spam tactics mutate in this ecosystem: days, weeks?