DL0134 Interleaved Vision-Language Architectures

An interleaved vision-language architecture flattens a document into one sequence that alternates text spans and images, projects each image’s patch features into the language model’s embedding space, and trains a single decoder with ordinary next-token prediction over the mixed stream. This is early fusion: visual tokens occupy real sequence positions, so they are keys and values for every text token in every layer, and text tokens are keys and values for the image. The alternative family, cross-attention conditioning (Flamingo, Idefics1), keeps vision outside the text sequence: a frozen ViT plus a Perceiver resampler compresses each image to a fixed latents, and newly inserted gated cross-attention layers let the frozen LM read those latents without ever lengthening its own sequence. The consequences are structural rather than cosmetic. Early fusion makes cost quadratic in the joint length and demands full multimodal pre-training, but it gives uniform bidirectional-within-causal mixing, unlimited image ordering, and (with discrete VQ image tokens) the ability to generate images from the same head. Cross-attention decouples image count from the text sequence and protects a strong frozen LLM, at the price of a hand-designed conditioning path, per-image attention masks that weaken multi-image relational reasoning, and no route to image output.

(1) Where Fusion Happens: early fusion concatenates projected patch embeddings into the token stream, so mixing occurs in every self-attention layer; cross-attention confines mixing to a handful of inserted layers reading a fixed latent set.
(2) Sequence Length Is The Cost Model: a ViT-L/14 at 336 px yields tokens per image, so eight images add 4608 positions and roughly the attention cost of a text-only forward pass.
(3) Attention Pattern And Multi-Image Reasoning: Flamingo masks each text token to cross-attend only to the most recent preceding image, while early fusion lets any token attend to all earlier images and their surrounding text.
(4) Parameter Budget And Text Regression: cross-attention adds new trainable blocks around a frozen LM and preserves text-only quality by construction; early fusion updates the whole backbone and risks catastrophic forgetting unless text data is replayed.
(5) Generation Symmetry: only the interleaved token-in-sequence form can emit images, because image outputs must live in the same vocabulary the head predicts (Chameleon, Emu3).
(6) Data Requirement: both families need interleaved web corpora such as M3W or OBELICS rather than caption pairs, since few-shot in-context learning is what interleaving buys.

Figure 1: The two conditioning paths. In early fusion the image becomes sequence positions inside the shared stack, so every layer mixes modalities and the sequence grows with each image. In cross-attention conditioning the image stays outside the text sequence as a fixed set of latents, read through gated layers whose gate starts at zero so the frozen LM is initially unchanged.

Practice has converged on early fusion, and the reason is mostly negative: the cross-attention path is an extra design surface with its own hyperparameters (how many layers to insert, resampler depth, latent count, gate schedule) and it caps the visual information at latents regardless of image resolution, which destroys dense OCR and chart reading. Ablations in MM1 found the interleaved token-in-sequence recipe matched or beat cross-attention at equal scale while being simpler, and the lineage of the same lab’s open models makes the shift explicit: Idefics1 (Flamingo-style cross-attention) → Idefics2 (early fusion with pooled visual tokens). Early fusion then imports the cross-attention family’s one real advantage, token compression, without its plumbing: Idefics3 pixel-shuffles patch blocks into one token, Qwen2-VL merges adjacent patches in the projector, and both keep the compressed tokens in the main sequence. What remains genuinely hard in early fusion is positional encoding for a 2D object embedded in a 1D stream, handled by explicit row-separator tokens in Fuyu or by multimodal rotary embeddings that give an image token separate height, width, and time indices.

Mathematical Formulation:

Where:

• are the image height and width and the patch size, so is the visual tokens per image: at this is .
• is the number of text tokens, the number of images in the document, and the joint sequence length the early-fusion decoder actually processes; is the model width.
• is the self-attention cost, quadratic in , so each added image contributes both its own block and cross terms with all existing tokens.
• is the raw image, the resampled latents, and the latent count ( in Flamingo), which is why is linear in and leaves the LM’s own untouched.
• are the frozen LM hidden states at layer and a learned scalar gate initialized to zero, so makes the inserted block an identity at step 0 and training degrades the language model gradually rather than abruptly.

Figure 2: Why token count dominates the choice. Early fusion pays in the joint sequence, so eight full-resolution images cost roughly a text-only pass, while cross-attention stays near because its cost is linear in . Patch pooling recovers most of the gap without leaving the sequence, which is the modern compromise.