DL0115 Visual Chain-of-Thought

Text-only CoT helps a vision-language model because it lets the model spend more tokens on intermediate reasoning, but it cannot help with detail the vision encoder never tokenized. A CLIP ViT-L/14 at produces a fixed grid of 576 patch tokens, so a photograph is downsampled by x before a single transformer layer runs and a 60-pixel license plate lands inside less than one patch. Visual CoT closes that gap by making localization an explicit step in the reasoning chain: the model first emits a bounding box for the region the sub-question depends on, that crop is re-encoded at the encoder’s native resolution, the new visual tokens are appended to the context alongside the original global tokens, and only then does the model answer. The chain becomes localize → crop → re-encode → reason, and it can iterate, which is exactly what multi-step spatial questions need: “what does the sign to the left of the red truck say?” decomposes into finding the truck, resolving left of in the global view, then reading text at a magnification where the glyphs actually exist. The VisCoT work trains this behavior with 438k question-answer pairs carrying intermediate bounding-box annotations, and reports its largest gains on text-rich and small-object splits, precisely where fixed-resolution encoding destroys the evidence.

(1) Perception Bottleneck, Not Reasoning Bottleneck: on high-resolution inputs the failure is usually that the target occupies a fraction of a patch, so extra reasoning tokens cannot recover it.
(2) Crop As A Resolution Amplifier: re-encoding a crop at raises the sampling density on that region by without changing the encoder.
(3) Global Tokens Must Be Retained: a crop destroys the frame of reference, so relational predicates such as “left of” or “third from the top” require keeping the original 576 tokens in context.
(4) Errors Compound Multiplicatively: the answer is correct only if localization and reading are both correct, so a 0.9 localization accuracy caps a two-hop chain at 0.81.
(5) It Needs Grounding Supervision: unlike text CoT, the intermediate step is a box, so it comes from bbox-annotated data, a detector tool, or a localization reward rather than from prompting alone.
(6) Sequential Cost: each hop is an extra forward pass over the encoder plus a longer prefill, which trades latency for accuracy in a way tiling does not.

Figure 1: Localization becomes an explicit reasoning step: the crop is re-encoded at native resolution and concatenated with the global tokens, so each hop of a multi-step spatial question is answered at the magnification it needs while the frame of reference survives.

The cost profile is what makes this a design decision rather than a free win. Visual CoT buys resolution with sequential compute: two or more encoder passes and a growing prefill, so at fixed batch size it roughly doubles time-to-first-token, and every hop is a place where a bad box silently poisons everything downstream. Tiling approaches such as AnyRes buy resolution with parallel compute instead, encoding a fixed grid of sub-images in one pass, which is friendlier to serving but pushes the visual context to 2,880 tokens or more and makes the language model’s attention the new bottleneck. The interesting property of the crop-based chain is that it is adaptive: it spends its extra tokens only on the region the question actually depends on, which is why it scales to K screenshots and document scans where uniform tiling would need dozens of tiles.

Mathematical Formulation:

Where:

• is the number of visual tokens per encoder pass, the encoder’s square input side, and the patch size; gives the familiar 576 tokens.
• are the original image dimensions and the predicted crop, both in pixels, with the crop constrained to lie inside the image.
• is the areal sampling gain on the cropped region, so the linear magnification is for the numbers above.
• is the probability the emitted box actually contains the target and the probability of reading it correctly once magnified; is end-to-end accuracy for a two-hop chain.
• For a -hop chain the same argument gives , and total visual tokens grow as because the global view is kept.

Figure 2: A 60-pixel target inside a image gets under one patch from a single low-resolution pass and about four from tiling, but roughly 92 patches after a crop is re-encoded, at half the token cost of tiling.