DL0133 VLM Fusion: Cross-Attention, Q-Former, and MLP

All three designs answer one question: where does the frozen vision encoder’s patch grid meet the language model? Linear/MLP projection (LLaVA, LLaVA-1.5, Qwen2-VL, InternVL) maps every patch embedding into the LLM’s token space with a two-layer MLP and concatenates the result onto the text sequence, so vision and language share one self-attention stack. Perceiver Resampler / Q-Former (Flamingo, BLIP-2) inserts a small cross-attention module in which a fixed set of learned queries (64 latents in Flamingo, 32 in BLIP-2) attends over the patch tokens and emits exactly vectors, decoupling the LLM’s sequence length from image resolution. Gated cross-attention fusion (Flamingo’s xattn-dense blocks, Llama 3’s vision adapter, NVLM-X) never puts image tokens in the LLM sequence at all: new cross-attention layers are interleaved into the language stack (every fourth layer in Llama 3) and read the patch tokens as an external key/value memory, with a tanh gate initialized to zero so the model starts out behaviorally identical to the text-only LLM. The trade-off is compute versus fidelity versus intrusiveness: MLP concatenation is the simplest and preserves the most visual detail but makes prefill grow as ; resampling is cheapest and constant-cost but imposes a hard information bottleneck; cross-attention keeps text throughput almost untouched and protects a frozen LLM, at the price of new parameters and a more complex training recipe.

(1) Where The Visual Tokens Live: MLP projection puts them inside the LLM sequence, a resampler puts a compressed of them inside, and cross-attention keeps them outside as an external memory.
(2) Prefill Cost Scaling: concatenation pays per layer, a resampler pays with , and cross-attention pays only in the inserted layers.
(3) Added Parameters: LLaVA-1.5’s connector is roughly 20M parameters of MLP, BLIP-2’s Q-Former is about 188M with a BERT-base initialization, and Llama 3’s cross-attention adapter adds tens of billions at the 405B scale.
(4) Information Bottleneck: a fixed caps how much of a high-resolution image can survive, which is why resampler-based models underperform on dense OCR, charts, and counting.
(5) Frozen Versus Tuned Backbones: the Idefics2 ablation found cross-attention wins when the LLM is frozen, while the fully autoregressive concatenation design wins once the LLM is unfrozen.
(6) Token Count Beats Connector Design: MM1’s ablations show the connector type matters far less than image resolution and visual token count, which is why the field converged on MLP plus cheap token compression.

Figure 1: The three fusion families differ only in where the patch tokens are allowed to go. Concatenation admits all of them into the shared sequence, a resampler admits a fixed , and gated cross-attention admits none, reading them instead as an external memory through layers whose tanh gate starts at zero.

The training recipe follows from the architecture. An MLP connector is so small that a two-stage schedule with about 558K caption pairs for alignment and 665K instruction samples for tuning is enough to reach state-of-the-art benchmark scores, which is what made LLaVA-1.5 reproducible on a single node. A Q-Former is a real transformer that must learn what to query, so BLIP-2 needs a dedicated representation-learning stage (contrastive, matching, and captioning objectives) before the generative stage, and remains the least data-efficient of the three per unit of final accuracy. Cross-attention sits in between: the gate makes optimization stable and lets you keep the language model frozen, so text benchmarks cannot regress, but you are training new layers that must learn to be useful without ever seeing visual tokens in the residual stream. NVLM’s controlled comparison is the cleanest evidence on the trade-off, finding the decoder-only concatenation variant stronger on OCR and multimodal reasoning while the cross-attention variant gave better throughput on high-resolution inputs.

Mathematical Formulation:

Where:

• is the input image, the vision encoder (typically a frozen CLIP or SigLIP ViT), and its patch token grid.
• is the projected visual sequence, with the connector weights and a GELU nonlinearity; a single recovers the original linear projection of LLaVA-1.
• is the patch count ( for ViT-L/14 at 336 px), the text length, the LLM sequence length, the LLM width, and the vision width.
• are the learned latent queries and the resampled output; is fixed at design time, so has the same size for a thumbnail and for a 4K page.
• is the text hidden state inside the LLM and the scalar gate parameter, initialized so that and the pretrained language behavior is exactly preserved at step zero.
• is the per-layer self-attention cost that concatenation inflates, and the per-layer cost of a cross-attention block, which is linear in rather than quadratic.

Figure 2: Prefill attention cost relative to a text-only forward pass at . At visual tokens, concatenation costs about the text-only baseline because the whole stack pays , gated cross-attention costs about since only 8 of 32 layers see the image and they scale as , and a 64-latent resampler stays near because the LLM never sees more than positions.