DL0135 Dynamic High-Resolution ViT

A pretrained vision tower expects one fixed square input, usually or , with exactly one learned position embedding per patch. Squashing a receipt or slide into that square applies anisotropic scaling (0.30x horizontally against 0.90x vertically), which shears glyphs and bar charts before the encoder sees a single pixel, while interpolating the position-embedding grid up to native resolution is both off-distribution and quadratically expensive in attention. Dynamic high-resolution slicing, the AnyRes family used by LLaVA-NeXT, UReader, Monkey, MiniCPM-V and InternVL 1.5, avoids both by never changing the encoder’s input size: it picks a tile grid whose aspect ratio best matches the image, resizes the image to exactly , cuts it into native-resolution tiles that are encoded independently, and appends a downsampled full-image thumbnail to restore the global layout that slicing destroys. Because the target width and height are both integer multiples of the same tile size chosen to match the input ratio, the horizontal and vertical scale factors are (nearly) equal, so the resize is isotropic and no geometry is distorted. What remains is a budgeting problem: the token count grows linearly in tiles, so token compression and a cap on the tile product decide how much resolution you can actually afford.

(1) Aspect-Ratio Matched Grid: enumerate all grids with and pick the one minimizing the log-ratio mismatch against , which is exactly the quantity that becomes anisotropic shear if you get it wrong.
(2) Tiles Stay At Native Resolution: every tile is precisely , so the frozen CLIP or SigLIP tower runs at its pretraining resolution with no position-embedding interpolation and no distribution shift.
(3) Thumbnail Carries Global Context: tiles are encoded independently, so no attention crosses a seam; a low-resolution whole-image view is concatenated to supply page layout and object-scale cues.
(4) Linear Instead Of Quadratic Attention: encoding tiles of patches costs rather than for one giant grid, a -fold saving that is why slicing scales to 4K pages.
(5) Token Compression Is Mandatory: a pixel-unshuffle of stride 2 folds each patch neighborhood into one embedding, cutting 1024 patches per 448 tile to 256 tokens; without it a 12-tile page costs over 13k tokens.
(6) The LLM Must Be Told The Layout: the flattened tile sequence is ambiguous, so implementations insert row separator tokens or 2D tile indices, otherwise a and a arrangement look identical downstream.

Figure 1: The reference circle is the whole story. A single square resize applies different scale factors per axis, so the circle becomes an ellipse and every glyph shears; the aspect-ratio-matched grid scales both axes by 0.90x, keeps each tile at the encoder’s native , and pays for the lost cross-tile view with one global thumbnail.

Two details separate a correct implementation from a leaky one. First, the resize to is only exactly isotropic when the chosen grid ratio equals the image ratio; otherwise a residual anisotropy survives, equal to the log mismatch, and it is bounded by how fine your candidate set is. With a 4:3 photo still carries 0.29 of residual log-anisotropy because the exact grid needs 12 tiles, which is why InternVL 1.5 raises to 12 and InternLM-XComposer2-4KHD pushes to 55 tiles, and why some systems pad to the grid instead of stretching. Second, slicing changes what the encoder can see: a table row spanning a vertical seam is split across two independently encoded tiles, so the fusion must happen inside the language model rather than in the vision tower, and this is the main reason answers to fine-grained OCR questions degrade near tile boundaries. Token cost is the binding constraint in production: LLaVA-NeXT at with at most 4 tiles plus a base view spends tokens per image, and InternVL 1.5 with 12 tiles plus a thumbnail spends only because pixel shuffle compresses each tile fourfold.

Mathematical Formulation:

Where:

• is the selected tile grid, and the candidate set of grids whose tile product stays within the budget .
• and are the original image width and height, and is the encoder’s native side length (336 for LLaVA-NeXT, 448 for InternVL 1.5).
• and are the per-axis scale factors of the resize, and is the residual anisotropy: means the resize is isotropic and geometry is preserved exactly.
• is the vision-token count fed to the LLM, with the accounting for the global thumbnail.
• is the patch size (typically 14) and the pixel-shuffle stride, so gives tokens per tile.
• The objective is stated in log space because minimizing is identical to minimizing , which makes aspect matching and distortion control the same problem.

Figure 2: The two knobs pull against each other. Raising is what drives residual anisotropy to zero for awkward ratios (left), but each extra tile buys a fixed block of tokens, so only pixel-shuffle compression keeps a 12-tile page inside a realistic per-image token budget (right).