DL0141 Visual Token Compression for OCR

The techniques that survive contact with documents all compress along the channel axis rather than the token axis: a pixel unshuffle (InternVL) or a strided convolutional reducer (DocOwl 1.5) folds each block of patch embeddings into one vector of width and projects it back to , so four patches become one token while the patch-to-region mapping stays bijective and no pixel is discarded. The second half of the recipe is that the token budget must keep scaling with input resolution: dynamic tiling and native-resolution patching (Qwen2-VL) give a 1275×1650 scan roughly 2,600 tokens of 28×28 pixels each, whereas a fixed- resampler hands the same page 64 tokens no matter how many glyphs it contains. What actually kills OCR is rarely the compressor itself but the resize step in front of it: squeezing a page into 336×336 makes an 11 pt glyph thinner than one 14-pixel patch, and no downstream module can recover strokes the encoder never sampled. The useful mental model is glyph density: keep the number of glyphs covered by a single visual token near one, and compression is nearly free; push it toward ten and character-level accuracy falls off a cliff while scene-level captioning barely moves.

(1) Compress Channels, Not Positions: pixel unshuffle and conv reducers move information into the feature dimension, so a length reduction still lets every output token point at a known rectangle of the page.
(2) Keep The Budget Resolution-Dependent: a compression ratio is safe, a compression target is not; dynamic tiling plus native-resolution patching lets a dense page buy more tokens than a photo of a beach.
(3) Respect Glyph Nyquist: the binding constraint is stroke width versus patch size in the resized image, which is why 336-pixel inputs cap document accuracy regardless of the connector.
(4) Fixed-Query Resamplers Lose The Wrong Thing First: learned queries cross-attend to all patches without a positional index, so reading order and rare characters degrade before object-level semantics do.
(5) Two-Scale Views Are Cheap: a global thumbnail supplies layout while local tiles supply glyphs, which is why AnyRes-style designs use tokens rather than one giant grid.
(6) Prune Late And Query-Aware: dropping half the visual tokens after LLM layer 2 (FastV) saves about 45% of prefill FLOPs on scene VQA but deletes whole text lines when the question has not yet been attended to.

Figure 1: Same patch grid, three compression axes. Only the top row keeps an exact mapping from output token back to page rectangle, which is what OCR decoding depends on; the middle row replaces that mapping with content-addressed slots, and the bottom row keeps positions but deletes evidence.

Why the distinction matters becomes obvious once you count information. Natural images are locally redundant, so averaging neighbouring patches costs almost nothing; a page of text is close to the opposite, since each glyph is a high-entropy symbol whose identity cannot be inferred from its neighbours and whose position carries the reading order. A learned resampler is a query-agnostic bottleneck: it must decide what to keep before the question arrives, and a fixed 64-slot budget forces it to summarise, which is exactly the wrong operation for text. Structured merging instead makes a bounded, uniform trade, and empirically the boundary sits near one glyph per token. The design lineage of production VLMs follows that logic directly: Q-Former resamplers → pixel unshuffle with dynamic tiles → native dynamic resolution with a 2×2 patch merger, each step trading a smaller guaranteed budget for a budget that grows with how much text is actually on the page.

Mathematical Formulation:

Where:

• is the patch count the vision encoder produces from a resized input of size with patch size , and is the number of tokens actually handed to the language model.
• is the spatial merge factor; pixel unshuffle with concatenates 4 patch embeddings into width and projects back to , so length drops with no averaging.
• is the original page resolution and the original pixels covered by one visual token, which is the honest measure of compression because the resize is itself a compressor.
• is the glyph count on the page and the glyphs per visual token; is the practical safety line for character-accurate reading.
• is the text prompt length and the model width, so prefill is quadratic in the combined sequence while the KV cache grows linearly.

Worked Example, One Dense A4 Page At 150 DPI:

Assuming about 3,200 glyphs on the page, an AnyRes layout of four 336-pixel tiles plus a thumbnail lands at roughly one glyph per token, while a single 336-pixel view lands at 5.6 and reads only headlines. The last line is the bill: those extra tokens cost 25 times the attention work in prefill, which is precisely why the compressor exists and why the interesting engineering is choosing the smallest that still keeps near 1 for the document class you serve.

Figure 2: Compression is only meaningful relative to glyph density. Configurations inside the band give each visual token roughly one character and read reliably; a 64-slot resampler asks one token to encode about 50 glyphs. Systems trained specifically for optical text compression can operate right of the band, but at a measured precision cost, and below the band extra tokens buy nothing.