DL0137 Early vs Late Fusion Multimodal

The two designs differ in where the modalities meet. Late fusion (the modular VLM recipe behind BLIP-2, LLaVA, and Qwen2-VL) keeps a separately pretrained vision encoder, compresses its output through a small connector (an MLP projector, a Q-Former, or gated cross-attention), and feeds the resulting soft tokens into a pretrained text LLM, so nearly all capacity is inherited and only the connector plus a light fine-tune is trained. Early fusion (native multimodal, as in Fuyu-8B, Chameleon, and Llama 4) discretizes or linearly projects raw patches into the same token stream as text and runs one shared transformer from layer 0, so every layer attends across modalities and the model can emit image tokens as well as text. Early fusion is representationally stronger and, per recent scaling-law work, not worse per FLOP, so its problems past 100B parameters are systems and data problems rather than a representational ceiling. You cannot reuse a trained text-only 100B checkpoint, a Chinchilla-scale run now wants roughly 2T interleaved tokens that do not exist at text quality, the shared dense weights must be split between modalities, and every image inflates the sequence that attention pays for.

(1) Meeting Point Defines The Family: late fusion joins modalities after a frozen encoder has already compressed the image; early fusion joins them at the token level, before any transformer layer.
(2) Checkpoint Reuse Is The Real Asymmetry: late fusion amortizes the trillions of text tokens already spent on the LLM, while a native 100B model pays that bill again on scarcer interleaved data.
(3) Modality Competition In Shared Weights: a dense early-fusion stack allocates the same parameters to pixels and text, so raising the image-token fraction costs text-benchmark quality and turns into a hyperparameter you cannot sweep cheaply at 100B.
(4) Token Budget Dominates The Context: one 512px image is about 1024 VQ tokens, so a four-image document spends 4096 tokens on pixels and quadratic attention absorbs the difference.
(5) Optimization Instability Grows With Width: heterogeneous token statistics drive logit growth and loss spikes, which is why Chameleon needed QK-Norm, reordered normalization, and dropout to keep a 34B run stable.
(6) Modularity Versus Capability: late fusion lets you swap the encoder or raise resolution for the price of a connector re-train, while early fusion buys interleaved any-to-any generation that a frozen tower cannot express.

Figure 1: Late fusion (left) is an assembly of pretrained parts, so the trainable surface is a projector of roughly 0.02B parameters and the alignment budget is on the order of 1B image-text tokens. Early fusion (right) is one homogeneous stack from random init, which is simpler to shard but means the whole 100B must be pretrained on interleaved data.

The scaling wall past 100B is mostly arithmetic. A dense 100B model trained compute-optimally wants about 2T tokens and roughly FLOPs, and in native early fusion those tokens must be interleaved image-text documents; the supply of such data at web-text quality is far smaller than the text corpus, so teams either repeat data or dilute the text share and watch reasoning benchmarks regress. Late fusion sidesteps this entirely because the expensive part is already paid: aligning a connector on a couple of billion tokens with a frozen backbone is three orders of magnitude cheaper than a from-scratch native run at the same parameter count. On top of the data problem, dense early fusion suffers capacity contention, since gradients from visual reconstruction and from language modeling compete for the same MLP weights, and sparsity is the standard fix: a mixture-of-experts stack with modality-aware routing gives each modality its own parameters while keeping active FLOPs fixed, which is why recent native models are almost always sparse. Early fusion does win on the systems side (no separate vision tower to shard, no encoder-LLM pipeline bubble, and a single tokenizer path), and it is the only option when the target task requires interleaved generation or genuinely fine-grained grounding that a pooled 576-token projection has already destroyed.

Mathematical Formulation:

Where:

• is the generated output; the first two equations are the late-fusion path and the third is the early-fusion path, where a single stack consumes the whole interleaved sequence.
• is the image and the text; is the pretrained vision encoder, the text embedding, and the connector, typically the only trained matrix during alignment.
• for are the interleaved tokens, either VQ image codes or linear patch projections placed in the same sequence as text tokens.
• is the parameter count, the training tokens, and the training FLOPs; is the Chinchilla compute-optimal ratio, which for demands about 2T tokens.
• is the layer count and the model width, so shows that image tokens entering the shared stack are charged quadratically, not linearly.
• is the image-token fraction of the pretraining mixture, so is the surviving text budget; at fixed , every point of is taken directly out of language modeling.

Figure 2: Left: from-scratch native training scales as under the compute-optimal token rule, while connector alignment scales linearly in , so the gap at 100B is roughly three orders of magnitude. Right: the number of tokens an image consumes spans about an order of magnitude across tokenizers, and in early fusion every one of those tokens enters the shared quadratic attention rather than being pooled away first.