Design an OCR system that converts millions of scanned documents (contracts, invoices, old books, fax-quality pages) into clean, structured text for a document platform like AWS Textract, Google Document AI, or Mistral’s OCR API. The output must preserve reading order, tables, and equations, not just raw characters, because the text feeds search indexes and RAG pipelines downstream.
How would you design this system? Cover the recognition approach, layout and structure handling, accuracy measurement, and the cost of processing at scale.
The Problem: a skewed, coffee-stained scan of a two-column contract must come out as correctly ordered, structured text, at millions of pages, at a cost a business can pay. Design the reader that never silently scrambles a page.
Answer
The design is a single-pass document VLM pipeline in the style of AI2’s olmOCR: lightweight preprocessing normalizes each page (orientation, deskew, denoise, blank rejection), then a fine-tuned 7B-class vision-language model reads the whole page in one pass and emits structured output (Markdown for headings, HTML for tables, LaTeX for math), and programmatic unit tests verify each page (does the table parse? does the equation recompile? is reading order sane?) before failures are retried or routed to a stronger model or a human queue. The two pivotal decisions are reading whole pages in one pass instead of a detect-then-crop-then-recognize cascade (which accumulates errors and loses reading order), and defining correctness as machine-checkable tests so both evaluation and training optimize what downstream consumers actually need.
(1) Single-Pass VLM Reader: a fine-tuned Qwen2.5-VL-class model turns a page image directly into structured Markdown/HTML/LaTeX, skipping the multi-stage cascade.
(2) Document Anchoring: born-digital PDFs contribute their embedded text and metadata into the prompt alongside the page image, grounding the model and cutting hallucination.
(3) Preprocessing Gate: orientation detection, deskew, denoise, and blank-page rejection run before the expensive model call.
(4) Verifiable QA: per-page unit tests (table parses, math recompiles, reading order monotonic); failures retry with different sampling or escalate to a stronger model or human review.
(5) Batch-First Serving: offline GPU batches with vLLM/SGLang; a tuned open pipeline processes a million pages for roughly $190, about 1/32 of frontier-API pricing.
Figure 1: The pipeline: cheap normalization first, one VLM read per page, then machine-checkable tests decide whether the page ships, retries, or goes to a human.
Clarify Before Designing:
(1) Document Mix: born-digital PDFs versus degraded scans; which languages; is handwriting in scope?
(2) Structure Fidelity: plain text, or RAG-grade structure with tables, equations, and reading order preserved?
(3) Volume and Latency: nightly millions in batch, or interactive upload-and-wait in seconds?
(4) Deployment Constraints: customer data residency or full self-hosting requirements (regulated archives)?
(5) Downstream Contract: does the consumer want Markdown, or structured JSON field extraction (invoice totals, dates) on top?
Leave a Reply