DL0138 Point-Informed Region VLM

A region-based VLM has to turn a geometric object (a box, a point, a scribble) into something a transformer can consume, and there are only three places to put it: the token sequence, the positional encoding, or the attention logits. The cheapest route serializes the geometry as coordinate tokens, either quantized bins added to the vocabulary (Kosmos-2 uses 1024 location tokens over a 32×32 grid, written like <loc_0512>) or plain digit strings (Shikra), so the box reaches self-attention as ordinary keys and values with zero architecture change. The second route extracts region features with RoIAlign or a learned sampler and projects them into the language model’s embedding space as extra tokens interleaved into the prompt (GPT4RoI, Ferret), which every text token can attend to like a word. The third route never adds tokens at all: it adds a bias matrix to before the softmax, so patches inside the referred region get a boost and patches outside get suppressed or hard-masked to . A fourth, architecture-free trick draws the prompt in pixel space (ViP-LLaVA’s overlaid arrows and circles, Set-of-Mark’s numbered masks) and lets the vision encoder’s own self-attention pick it up, which is the only option when the model is a frozen API. The choice is a trade among spatial precision, sequence growth, and how much grounding supervision you can afford to train on.

(1) Three Injection Points: geometry enters as tokens, as positional/coordinate embeddings, or as an additive term inside the attention logits, and most systems combine at least two.
(2) Coordinate Tokens Are Free But Quantized: a 1000-bin normalized grid on a 1024 px image gives roughly 1 px resolution, but the same bins on a 4K crop after tiling lose fidelity and shift under aspect-ratio padding.
(3) Region Tokens Buy Content, Not Just Location: a pooled RoIAlign vector carries appearance of the referent, so the model can describe a region it could not have isolated from coordinates alone.
(4) Attention Bias Adds Zero Parameters: is added elementwise to the pre-softmax scores, leaving the cost and the KV cache unchanged, and a per-head can be learned.
(5) Hard Masks Destroy Context: setting outside the region removes exactly the surrounding evidence that relational questions need, and a fully masked row produces a degenerate softmax.
(6) Points Are Cheaper Supervision Than Boxes: Molmo’s pointing data shows a single pair is enough for counting and referring, and it avoids the box-regression ambiguity for deformable objects.
(7) Pixel-Space Prompts Work On Frozen Models: drawing the mark changes the image, not the architecture, but it occludes content and fails on thin or tiny structures.

Figure 1: Four ways a box reaches attention. Mechanisms (1) and (2) extend the token sequence so geometry becomes ordinary keys and values, mechanism (3) leaves the sequence untouched and edits the pre-softmax logits, and mechanism (4) modifies the pixels so the vision encoder does the work. Only (3) avoids sequence growth entirely.

The region-feature path is the one that most resembles classical detection heads: pool the backbone feature map over the box (RoIAlign → linear projector → language-model token), then concatenate a coordinate embedding so the model knows where the appearance came from, not only what it looks like. Ferret generalizes this with a spatial-aware visual sampler that samples and pools features inside an arbitrary free-form shape, so a point, a box, and a scribble all reduce to the same fixed-size token, and Groma pushes localization into the tokenizer itself by proposing regions up front and giving each one a referable ID token. The attention-bias path is complementary and is the literal reading of the question: because the softmax is invariant to nothing but its own inputs, a single additive term reweights how much probability mass a text query spends on patches inside versus outside the referent, and a moderate concentrates mass without deleting surrounding context. In practice, training matters more than the plumbing: none of these mechanisms produces reliable grounding without a large corpus of region-text pairs, which is why grounded VLMs are trained on visual genome, RefCOCO-style referring expressions, and synthetically generated region captions.

Mathematical Formulation:

Where:

• is the region token appended to the prompt, and is the full input sequence of text, region, and image-patch embeddings.
• is the vision-encoder feature map, the box in pixels, the image side used for normalization, and the normalized coordinate vector.
• projects the pooled region feature into the language-model width and embeds the coordinates; both are the only new parameters this path needs.
• are the query and key projections of , the per-head dimension, and the attention matrix over all positions.
• is the additive region bias, the set of positions that query is encouraged to attend to (the patches overlapping the referred box), and the per-head bias strength; recovers plain attention and gives a hard mask.
• is the number of visual plus text tokens, the number of added region or coordinate tokens, and the model width, so token-based injection is quadratic in while the bias term is free.

Figure 2: The same query, the same keys, three values of . A soft bias raises in-region mass from a small baseline to the majority while preserving outside context, which is what relational questions need; hard masking reaches 100% in-region mass and throws that context away.