Design a RAG-based customer support assistant for a company like Stripe, Shopify, or a large telecom. The assistant is grounded in the company’s own knowledge base: help articles, product documentation, past resolved tickets, and policy pages. It must answer accurately, cite its sources, refuse to fabricate when the answer is not in the knowledge base, escalate to a human when the question is high-stakes (billing, account access, legal) or when its confidence is low, and stay current as the knowledge base is edited every day.
The cost structure is asymmetric: a hallucinated refund policy or a confident wrong answer about account access is far worse than admitting ignorance and handing the conversation to an agent. A wrong answer can create a chargeback, a regulatory complaint, or a churned account, while an escalation costs a few dollars of agent time.
How would you design this system? Cover the retrieval architecture (chunking, embedding, hybrid search, re-ranking), the generation and grounding strategy, the confidence and safety gating that decides when to answer versus escalate, the human handoff path with full context, and how you evaluate answer quality and hallucination rate.
The Problem: the answer either exists somewhere in a knowledge base that changes daily, or it does not exist at all, and the system has to tell those two cases apart before it opens its mouth.
Answer
The design is a retrieve, re-rank, generate pipeline wrapped in an explicit answer policy. Hybrid retrieval over structure-aware chunks pulls 50 candidates, a cross-encoder re-ranker trims them to 8, and the generator may only assert what a cited passage actually says. Two decisions carry the answer. First, abstention and escalation are first-class outputs: a risk classifier runs before generation, a retrieval-confidence threshold guards coverage, and a per-claim groundedness verifier checks the draft, so the default under doubt is a warm human handoff rather than a fluent guess. Second, the index is versioned and refreshed nightly by delta, with product, plan, and region metadata as hard filters, because most support “hallucinations” are really retrieval of the wrong but plausible policy version.
(1) Structure-Aware Chunking: split on headings at roughly 400 tokens with 15% overlap, keep tables and code blocks intact, and prepend the article title and breadcrumb so an isolated chunk is self-describing.
(2) Rich Chunk Metadata: product, plan, region, locale, source type, version, and effective date travel with every chunk and act as hard filters before scoring.
(3) Hybrid Retrieval Plus Re-ranking: BM25 catches error codes, SKUs, and API field names; dense embeddings catch paraphrase; fusion gives 50 candidates and a cross-encoder picks the top 8.
(4) Citation-Constrained Generation: numbered passages in the prompt, sentence-level citation ids in a structured output, low temperature, and an explicit insufficient_context abstain option.
(5) Three-Gate Answer Policy: risk classifier, retrieval-confidence threshold, and claim-level groundedness check; any gate that fires routes to a human instead of shipping the text.
(6) Warm Handoff and Content Loop: the agent inherits the transcript, the exact passages used, the extracted entities, and a draft reply; the agent’s final answer becomes an eval label and, when the gap is real, a new article.
Figure 1: One pass per question: rewrite and risk-classify, retrieve and re-rank, generate only from the surviving passages, then let the gate decide whether the customer sees the text or an agent does.
Clarify Before Designing:
(1) Volume and Channels: how many chat sessions per day, how many turns each, which languages, and is the user authenticated when they ask?
(2) Quality Bar and Deflection Target: what accuracy floor must hold for answers we do give, and who signs off on the refusal rate that buys it?
(3) Knowledge Base State: how much content is written for customers versus internal agents, are policies versioned by region and plan, and are ticket resolutions quality-reviewed?
(4) Account Access: may the assistant read account state (plan, invoices, outage status) through tools, or only public docs? This changes both retrieval and the risk gate.
(5) Latency and Cost Budget: target first token, target full answer, and the per-conversation cost ceiling relative to an agent contact.
(6) High-Stakes Definition: which intents are legally or financially sensitive (billing disputes, cancellation, identity, data deletion) and what retention rules apply to logged conversations?
Leave a Reply