Design a question-answering system that extracts an answer from a large document collection given a user query using a deep reading comprehension model. A legal or enterprise search product must let users ask natural-language questions (“What is the termination clause in the NDA?”, “What were the Q3 revenue figures for the EMEA region?”) and return exact answer spans extracted from a corpus of millions of documents: contracts, filings, internal wikis, emails.
The system must first retrieve the relevant documents or passages from the corpus (the retrieval stage), then run a reading comprehension model over the retrieved passages to extract the answer span (the reader stage), and finally rank or aggregate answers when multiple passages produce candidates. The retrieval stage must handle lexical mismatch (the query says “termination” but the document says “cancellation”), the reader must handle multi-hop reasoning where the answer requires combining information from two passages, and the system must say “I don’t know” when no passage contains a confident answer rather than hallucinating.
How would you design this system? Cover the retriever architecture (sparse BM25 vs dense bi-encoder vs hybrid retrieval), the reader architecture (extractive span prediction vs generative reading comprehension vs T5/BART-style seq2seq), how you handle multi-hop questions, how you calibrate confidence for abstention, and how you evaluate end-to-end QA accuracy (exact match, F1, answerability detection).
The Problem: one natural-language question, ten million documents that rarely use the asker’s words, and a product that must quote the exact sentence or admit it cannot find one.
Answer
The design is a retrieve-rerank-read pipeline with an explicit abstention gate. A hybrid retriever (BM25 plus a dense bi-encoder, merged by reciprocal rank fusion) pulls about 100 passages from a permission-filtered index, a cross-encoder reranker keeps the top 10, and an extractive reader with a no-answer head scores candidate spans in each passage. Spans are aggregated across passages, and a calibrated confidence threshold decides between returning a cited span and answering “I don’t know”. Multi-hop questions run the same loop twice: the first hop’s reader output becomes a bridge entity that rewrites the query for a second retrieval. The pivotal decisions are hybrid rather than pure dense retrieval (synonyms and exact identifiers both matter in legal and financial text), extractive rather than generative reading (verbatim spans with citations are the product, and a span cannot hallucinate), and a threshold set on the risk-coverage curve rather than on a raw softmax.
(1) Passage Index with Metadata: split documents into 200-300 token passages with overlap, and store document type, date, region, and access-control lists alongside both the inverted index and the ANN index so filters apply at retrieval time.
(2) Hybrid Retrieval: BM25 catches exact identifiers (clause numbers, “Q3”, “EMEA”) while the dense bi-encoder catches paraphrase (“termination” versus “cancellation”); reciprocal rank fusion merges both lists into a top-100.
(3) Cross-Encoder Reranker: joint query-passage encoding narrows 100 → 10 passages and is the largest single quality gain per millisecond in the pipeline.
(4) Extractive Reader with Null Head: start and end logits plus a no-answer score per passage; identical spans from different passages pool their probability.
(5) Iterative Multi-Hop: when hop one yields a bridge entity but no final answer, the query is rewritten and retrieval runs again, capped at two hops to bound latency.
(6) Calibrated Abstention: the null-versus-span margin threshold is chosen on the risk-coverage curve at the product’s tolerated error rate, and abstentions fall back to classic search or a human.
Figure 1: The retrieve-rerank-read path with its two exits, a cited span or an honest abstention, and the dashed multi-hop loop that rewrites the query with a bridge entity.
Clarify Before Designing:
(1) Corpus and Freshness: how many documents and pages, in which languages and formats (scanned PDFs need OCR), and how quickly must a new filing become searchable?
(2) Load and Latency: queries per day, acceptable p95 latency for an interactive answer, and whether a slower two-hop path is tolerable for a minority of questions?
(3) Answer Form: is the product a verbatim span with a citation, or a synthesized paragraph; may the system quote two passages for one answer?
(4) Multi-Hop Share: what fraction of real questions need two passages, and is a bridge through an entity (a counterparty, a region, a fiscal quarter) the dominant pattern?
(5) Labels: do in-domain question-answer pairs exist (support tickets, FAQ pages, past research memos), and what annotation budget do domain experts have?
(6) Error Asymmetry and Access: how much worse is a confidently wrong clause than “I don’t know”, and must answers respect per-user document permissions?
Leave a Reply