DL0112 VLM Visual Grounding

Visual grounding asks the model to return the image region a phrase refers to, and current VLMs differ mainly in where the coordinates are physically produced. The dominant generalist recipe treats a box as text: coordinates are normalized and quantized onto a fixed grid (Kosmos-2 adds 1024 dedicated location tokens, Qwen-VL emits integers from 0 to 999 inside <box> markers, Qwen2.5-VL moved to absolute pixel values inside JSON) and the language model decodes them autoregressively under ordinary cross-entropy, so detection, referring expression comprehension, and grounded captioning all become one sequence task. The specialist recipe keeps a DETR-style decoder with learnable queries, a box regression head, and L1 plus GIoU losses, fusing text into the visual features and aligning region embeddings to word embeddings contrastively (MDETR, GLIP, Grounding DINO, Florence-2). A third, hybrid recipe uses an LLM hidden state as a prompt for an external mask or box decoder, as LISA does by feeding a <SEG> embedding to SAM. Whichever decoder is used, localization quality is mostly decided upstream: the connector has to hand the LLM patch tokens with their 2D layout and position information intact, at a resolution where the target object still covers several patches. 3D grounding extends the output to a 7 to 9 DoF box and introduces the real difficulty, depth and scale ambiguity, resolved either from point clouds (ScanRefer-style proposal-and-match pipelines) or monocularly with camera intrinsics, as in Cube-LLM.

(1) Coordinates As Text: a box becomes four discrete symbols on a normalized grid, so the only training signal is token cross-entropy, with no IoU-aware term and no set matching.
(2) The Connector Decides Localization: raster-ordered ViT patch tokens through an MLP projector preserve geometry, while abstractors that pool everything into a few dozen learned queries discard the spatial layout a box depends on.
(3) Resolution Beats Bin Count: quantization error at 1000 bins is sub-pixel, but a small object that occupies one patch cannot be localized tightly at any bin resolution, which is why dynamic resolution and tiling (AnyRes, Qwen2-VL M-RoPE) matter more than the tokenizer.
(4) Detection-Head Decoding: query-based decoders with region-word contrastive alignment still lead on tight-IoU metrics and emit hundreds of boxes in a single forward pass.
(5) Hidden State As Prompt: a referent or <SEG> token embedding can drive SAM or a 3D mask decoder, decoupling language reasoning from pixel-level decoding.
(6) 3D Adds Scale, Not Just Dimensions: monocular predictions are only consistent if camera intrinsics enter the model or the normalization, otherwise depth estimates do not transfer across datasets.
(7) Metrics And Supervision: RefCOCO family Acc@0.5 for 2D, [email protected] on ScanRefer for 3D, and every box target must be expressed in the same resized or tiled frame the encoder sees.

Figure 1: One shared perception stack, three places to produce coordinates. The path image → ViT patches → projector → LLM is identical; only the last stage differs, and the projector is where grounding is usually won or lost.

It is worth doing the arithmetic on the tokenizer, because interviewers often assume quantization is the limiting factor. With a 1000-bin normalized grid on a 1344 px side, the worst-case error per edge is about 0.67 px, far below what Acc@0.5 can detect. The real sensitivity is object size: a square object of side whose four edges each shift by has , so a 16 px object falls below 0.5 IoU once edges move about 3 px, while a 128 px object tolerates roughly 24 px. That is why generalist grounding scores jumped with higher effective input resolution rather than with finer coordinate grids: the earliest text-token generalists sat near 52 Acc@0.5 on RefCOCO val, Qwen-VL reached roughly 89, and Florence-2-L about 93, close to dedicated grounding detectors.

Mathematical Formulation:

Where:

• is the target box in the encoder’s resized pixel frame of side , and is the prediction.
• is the -th coordinate token and the number of bins (1024 for Kosmos-2 location tokens, 1000 for Qwen-VL integers).
• is the image, the referring expression, and the previously emitted coordinate tokens, so decoding is strictly autoregressive.
• is the worst-case quantization error per edge in pixels, which scales linearly with input side and inversely with bin count.
• weight the L1 and GIoU terms used by regression-head decoders; text-token models have no analogue of either.
• is a 3D box with center , extents , and yaw ; full 9 DoF variants add pitch and roll.
• is the camera intrinsic matrix and the projected image point; this constraint is the required initial condition for monocular 3D grounding, since without the depth and the extents trade off freely.

Figure 2: Left: a 1000-bin grid costs well under one pixel per edge even at high resolution, so coordinate tokenization is rarely the bottleneck. Right: the Acc@0.5 budget is roughly 3 px of edge error for a 16 px object, 6 px for 32 px, and 24 px for 128 px, which is why small-object grounding is an effective-resolution problem.