DL0142 OCR-Free Document VLM

A multi-stage pipeline turns a page into a text document before any reasoning happens: rasterize → detect text regions → recognize each crop → classify layout blocks → sort them into reading order → recover table cell structure → serialize to Markdown or HTML → feed a text LLM. An OCR-free Document VLM deletes that entire chain and treats the page as an image: a dynamic-resolution ViT encoder cuts the raster into 14×14 patches, a 2×2 pixel-shuffle merge collapses them into one visual token per 28×28 pixel block, and a decoder-only LLM attends over those tokens to emit the answer, the Markdown, or the HTML table directly. The consequence is that self-attention itself becomes the layout model: column boundaries, cell alignment, chart axes, and legend-to-series association are learned from pixels rather than reconstructed by six independently trained components whose errors multiply. What you gain is robustness on infographics and rotated or borderless tables, where reported scores such as Qwen2.5-VL-72B’s roughly 96 ANLS on DocVQA and roughly 87 on InfographicVQA are far out of reach for a serialized-text pipeline. What you lose is character-level coordinates, per-token confidences, and cheap per-page cost, because a single A4 page at 150 DPI already costs about 2,835 visual tokens and the prefill over them is quadratic.

(1) Pixels In, Structure Out: the model never sees a text layer, so scanned pages, screenshots, and born-digital PDFs with broken embedded fonts all take the identical path.
(2) Dynamic Resolution Tokenization: instead of squashing every page to 224×224, the encoder keeps native aspect ratio and resolution, so an 8 pt footnote survives as its own tokens rather than being blurred away.
(3) Attention Replaces The Reading-Order Module: a three-column paper needs no LayoutReader-style sorter, because the decoder learns column continuation the way a language model learns syntax.
(4) Tables As Generated Markup: structure recognition becomes ordinary autoregressive decoding of HTML or Markdown, which handles borderless and spanning cells but has no per-cell confidence.
(5) No Error Compounding, No Coordinates: the pipeline’s five or six stages multiply their error rates, while the VLM has one loss and one failure surface but cannot tell you where on the page an answer came from.
(6) Token Budget Is The Real Constraint: visual tokens grow with the square of DPI, so resolution, tiling, and page count trade directly against context and prefill.

Figure 1: The same pixels, two failure surfaces. The pipeline produces an intermediate text document with coordinates that any downstream model can consume and any auditor can overlay, at the cost of five models whose accuracies multiply. The VLM is one differentiable stack with one loss, and its output carries no character boxes at all unless the model was explicitly trained to emit them.

The three hard document classes fail differently. On multi-column PDFs, a pipeline’s mistake is almost never recognition, it is serialization: a two-column paper with a full-width figure caption in the middle gets flattened into interleaved half-sentences, and the LLM downstream has no way to recover the intended order because the evidence, the geometry, was discarded. On tables, borderless layouts and spanning header cells break rule-based and detection-based structure recognition, whereas a VLM trained on HTML targets can emit rowspan and colspan because it saw the whole grid at once. On infographics and charts, OCR returns a bag of strings with no relations, so “which bar is tallest” or “what does the dashed series do after 2021” is unanswerable from the transcript; this is precisely where the ChartQA and InfographicVQA gaps are widest. The pipeline still wins wherever the requirement is verbatim fidelity plus provenance, since a hallucinated digit inside a generated table cell is indistinguishable from a correct one, while a low-confidence OCR crop announces itself.

Mathematical Formulation:

Where:

• is the number of visual tokens the encoder emits for one page, which is what actually enters the LLM context.
• and are the rasterized page height and width in pixels ( for A4 at 150 DPI), and is the effective patch stride after a 2×2 merge of 14×14 patches.
• is the attention cost before the first output token, quadratic in and linear in model width ; the KV cache grows linearly, so a 20-page document is a memory problem as well as a compute one.
• is the per-page success rate of pipeline stage and the number of stages, with running detection, recognition, layout, reading order, and table structure.
• is the end-to-end page accuracy: five stages that each succeed 95% of the time leave only about 77% of pages fully clean, and this multiplicative compounding is the structural argument for a single-stage model.

Figure 2: Resolution is the cost knob. The same A4 page costs 2,835 visual tokens at 150 DPI and 11,214 at 300 DPI, roughly 4x the tokens and 16x the prefill FLOPs, while a serialized OCR transcript of that page stays near 800 tokens whatever the DPI. The 2×2 pixel-shuffle merge is what keeps a full page under the common 16,384-token per-image cap at all.