Design the routing layer for an AI assistant that chooses, per request, between a small cheap model and a large expensive one. ChatGPT’s GPT-5 does exactly this with a real-time router that picks between its fast and reasoning variants, and research frameworks like RouteLLM show a trained router can cut serving cost by over 80% while holding about 95% of the strong model’s quality.
How would you design this system? Cover what the router sees and how it is trained, how answer quality is guarded, the cost accounting, and how the system adapts as models are added, upgraded, or deprecated.
The Problem: most requests are easy, a few are hard, and sending everything to the biggest model burns money on “hello”. Design the gate that sends each request down the cheapest road that still answers it well.
Answer
The design is a trained per-request router: a tiny scorer reads the request and predicts the probability that the strong model’s answer beats the weak one’s, then sends the request to the cheapest model expected to clear the quality bar. This is routing, not cascading, so each request normally pays for exactly one model. The two pivotal decisions are to train the router on human preference data (which model’s answer people actually preferred, as in Chatbot Arena battles) rather than hand-built rules, and to expose an explicit cost-quality operating threshold as a product knob, with an escalation fallback so a routing mistake costs extra compute, not a bad answer.
(1) Difficulty Signals: query embedding and length, task type, tool needs, and conversation state; the router is BERT-class or smaller, ~1/1000 the cost of a strong call.
(2) Preference-Trained Router: learn P(strong model wins | query) from human preference battles, augmented with LLM-judged comparisons; route strong when P passes the threshold.
(3) Explicit Operating Point: the threshold is chosen on the measured cost-quality curve and recalibrated as the traffic mix drifts.
(4) Guarded Fallback: if the small model or a verifier flags low confidence, escalate to the large model; routing errors become cost, not bad answers.
(5) Model Registry & Shadowing: candidates live behind a versioned registry; new models are shadow-scored on live traffic before the router is retrained or recalibrated.
Figure 1: The routing path: one cheap score per request, one paid model call, and an escalation hatch when the cheap path doubts itself.
Clarify Before Designing:
(1) Candidate Set: two models or many; closed APIs, self-hosted weights, or both; is the real split fast-versus-reasoning rather than small-versus-large?
(2) Decision Granularity: per request, per message in a conversation, or pinned per session for style consistency?
(3) Quality Bar: which evaluation defines “good enough” (human preference, task success), and how much loss versus the always-strong policy is tolerable?
(4) Router Budget: how much latency and cost may the router itself add per request?
(5) Traffic Shape: what fraction of traffic is trivial chat versus hard reasoning today, and how fast does that mix drift?
Leave a Reply