VLM Archives - Page 3 of 3 - Interview for Machine Learning

How do you align image tokens with text tokens in a vision-language model, for example in a system like LLaVA or Qwen2-VL?

Answer

Alignment is not one operation but three that have to agree: a representation map, a sequence layout, and a training curriculum. The dominant recipe runs a frozen contrastive vision encoder over the image, then a small trainable connector projects every patch embedding into the LLM’s token embedding space so the image becomes a block of ordinary tokens occupying real sequence positions. A CLIP ViT-L/14 at $336 \times 336$ produces $24 \times 24 = 576$ patch embeddings of width 1024, and a two-layer MLP maps them to the LLM width (4096 for a 7B decoder); the tokenizer’s single image placeholder is expanded in place into those 576 vectors. From that point alignment is learned by plain next-token prediction: text tokens read image positions through the same causal self-attention, and the gradient of the caption loss is what teaches the connector which visual direction means “a red bus”. The competing topology keeps images out of the sequence entirely and injects them through gated cross-attention layers inserted into a frozen LLM, as in Flamingo and Llama 3.2 Vision, trading sequence length for extra parameters.

(1) Shared Embedding Space: the connector’s only job is to land visual features in the same $d_{\text{model}}$ -dimensional space the token embedding table lives in, which is why an MLP with roughly 20M parameters is enough.
(2) Placeholder Expansion: the prompt carries one image placeholder id, and at embedding time that row is replaced by the $N$ projected vectors, so nothing downstream of the embedding layer knows the difference.
(3) Positional Alignment: patches are flattened row-major, so a purely 1D RoPE makes a vertical neighbor 24 positions away; 2D or multimodal RoPE restores height and width structure explicitly.
(4) Two Injection Topologies: prefix tokens with full self-attention are simple and preserve fine detail; gated cross-attention keeps the text stream’s length untouched and protects text-only ability.
(5) Staged Curriculum: stage 1 tunes only the connector on image-caption pairs with a frozen encoder and LLM, then stage 2 unfreezes the LLM on instruction data.
(6) Token Budget Is The Real Constraint: prefill attention is $O(L^2)$ in total sequence length, so pixel unshuffle, learned query resamplers, and tiling policies exist purely to control $N$ .

Pipeline diagram: a 336 by 336 image is encoded by a frozen ViT-L/14 into 576 patch features of width 1024, an MLP connector projects them to 4096 dimensions, and the resulting image tokens are spliced into the LLM token sequence between text embeddings before a causal decoder stack

Figure 1: The connector is the only trainable component in stage 1; after projection the 576 image tokens are indistinguishable from text embeddings to the decoder, which fuses both streams in a single causal self-attention stack.

Two details decide whether the alignment survives at scale. The first is resolution: a fixed $336 \times 336$ view destroys the small text and thin structures that document and chart questions depend on, so production models either tile the image into crops plus a downscaled thumbnail (AnyRes) or feed native resolution with a variable token count. The second is cost: because prefill attention grows quadratically, an AnyRes scheme with four crops plus a thumbnail spends 2,880 image tokens and roughly 25x the attention work of one 576-token view, before a single text token is generated. Compression is the usual answer, either a $2 \times 2$ pixel unshuffle that folds four patches into one channel-concatenated token or a query-based resampler that squeezes any number of patches into 32 or 64 learned slots. Both save compute by throwing away exactly the spatial precision that grounding and OCR need, which is why token budget, not connector architecture, is where most VLM design arguments actually land.

Mathematical Formulation:
$Z_v = g_{\phi}(x_{\text{img}})$
$H_v = W_2\,\sigma(W_1 Z_v)$
$N = \frac{H W}{P^2 r^2}$
$N = \frac{336 \cdot 336}{14^2 \cdot 1} = 576$
$E = [\, e(t_{1:k}),\; H_v,\; e(t_{k+1:m}) \,]$
$\mathcal{L} = -\sum_{i} \log p_{\theta}(t_i \mid t_{1:i-1}, H_v)$

Where:

$x_{\text{img}}$ is the image and $g_{\phi}$ the frozen vision encoder, giving patch features $Z_v \in \mathbb{R}^{N \times d_v}$ with $d_v = 1024$ for ViT-L.
$W_1, W_2$ and the nonlinearity $\sigma$ form the MLP connector, producing $H_v \in \mathbb{R}^{N \times d_{\text{model}}}$ in the LLM embedding space.
$H, W$ are image height and width, $P$ the patch size, and $r$ the pixel-unshuffle factor ( $r = 1$ means no merging, $r = 2$ cuts tokens 4x).
$e(\cdot)$ is the text embedding lookup, $t_{1:m}$ the prompt and response tokens, and $k$ the position of the image placeholder that $H_v$ replaces.
$E$ is the assembled input of length $L = m + N - 1$ , and $\mathcal{L}$ is the autoregressive loss masked to response tokens only, so no loss is computed on image positions.
Required initial condition: $g_{\phi}$ must already be language-aligned (CLIP or SigLIP pretraining), otherwise stage 1 has to learn the semantics as well as the projection.

Bar chart of image token counts and relative prefill attention cost for four vision-language configurations: 576 tokens at 1x, 1280 tokens at 4.9x, 1536 tokens at 7.1x, and 2880 tokens at 25x

Figure 2: Higher-resolution alignment policies buy detail with sequence length, and since prefill attention is $O(L^2)$ , moving from a single 576-token view to a five-crop AnyRes layout costs about 25x the attention work per prompt.

Property	MLP connector (prefix tokens)	Query resampler (Q-Former, Perceiver)	Gated cross-attention
Where fusion happens	In the input sequence, before layer 1	In the connector, then in the input sequence	Inside the decoder, at interleaved new layers
Tokens added to the LLM sequence	One per patch, 576 to 2,880 typical	Fixed, 32 to 64 regardless of resolution	Zero
New parameters	Smallest, roughly 20M for a 7B model	Moderate, a small transformer plus queries	Largest, extra attention plus tanh gates in the decoder
Prefill and KV-cache cost	Grows quadratically with token count	Nearly flat in image resolution	Linear in image features, no text-side growth
Typical failure mode	Latency and context blowup on multi-image or video prompts	Information bottleneck: weak OCR, dense counting, fine grounding	Harder to train, weaker at precise pixel-level reference
Text-only ability	Can regress once the LLM is unfrozen	Same risk if the LLM is tuned	Preserved by construction with a frozen LLM and zero-init gates

Tag: VLM

DL0110 VLM Image-Text Token Alignment