Author: admin

Temporal visual grounding (also called temporal sentence grounding or moment retrieval) maps one natural-language query onto a single interval inside an untrimmed video, and almost every difficulty follows from one mismatch: the metric is far more precise than the signal you can afford to look at. Benchmarks report R@1 at IoU 0.7 against a human-drawn span, yet the target moment is often a few percent of a video that runs for minutes, its boundaries are genuinely ambiguous (annotators disagree by seconds about when “opens the fridge” starts), supervision is a single positive span among thousands of candidate windows, and compute forces you to subsample frames so hard that the achievable temporal resolution can be coarser than the tolerance the metric grants. Worse, the standard datasets carry strong temporal location priors, so a model that never looks at the video can score respectably and mask the fact that no grounding is happening at all. Cross-modal features add their own problem, because image-level encoders describe objects well and describe verbs, ordering, and completion poorly, which is exactly the information a query like “right after he sits down” depends on.

(1) Tolerance Scales With The Moment, Not The Video: at a correctly sized 4-second moment tolerates a rigid boundary shift of only 0.71 s, so short moments demand sub-second precision while long ones are almost free.
(2) Boundary Ambiguity Lives In The Labels: event onsets and offsets are not crisp, so inter-annotator IoU sits well below 1.0 and an L1 boundary regression fits annotation noise past a certain accuracy.
(3) Sampling Stride Trades Precision Against Cost: sampling at fps sets a hard error floor near seconds, but keeping high makes large and early cross-modal fusion costs .
(4) Features Are Weak On Temporal Semantics: frozen CLIP-style per-frame embeddings encode appearance, so verbs, order, and “before/after” relations are nearly invisible without motion-aware or video-pretrained features.
(5) Location Priors Let Blind Models Win: on Charades-STA and ActivityNet Captions the ground-truth spans cluster in predictable places, so query-only or prior-only baselines are competitive and out-of-distribution splits expose the gap.
(6) Supervision Is Sparse And Assumes One Span: a single positive interval creates extreme foreground/background imbalance, and queries matching several disjoint intervals or none at all break single-span heads outright.

Figure 1: The tolerance the metric grants is proportional to the moment duration, so a strict threshold on short moments asks for a precision that the frame sampling stride cannot deliver. Below the dotted lines, the demanded accuracy is finer than one sampled frame, and no amount of head tuning recovers it.

Mathematical Formulation:

Where:

• is the annotated span and the prediction, both in seconds; and are the 1D intersection and union, and the handles disjoint spans.
• is the ground-truth duration, the IoU threshold, and the largest rigid shift of both boundaries that still passes, obtained from .
• indexes queries and is the usual R@1, IoU=m metric, a hard indicator that is flat inside the tolerance band and gives no gradient-like signal about how close a miss was.
• normalized by video length, and the loss weights; the L1 term is scale-sensitive and the IoU term is scale-free, which is why both appear in DETR-style grounding heads.
• is the video duration, the sampling rate, and the number of frame tokens; a 20-minute video at gives , and self-attention over it costs per query.

The design space has moved through three families and is now entering a fourth: sliding-window proposal ranking (dense candidate spans scored against the query, as in 2D-TAN), proposal-free span prediction (per-frame start and end distributions, as in VSLNet), DETR-style set prediction (learnable moment queries plus Hungarian matching with an L1 and IoU loss, as in Moment-DETR and CG-DETR), and video LLMs that emit timestamps as text or as dedicated time tokens. Each family inherits the same structural problems in a different shape: proposal methods make the imbalance explicit and cap resolution at the window grid, span prediction is cheap but assumes exactly one contiguous answer, set prediction handles multiple moments and a no-moment class but needs enough data to learn the matching, and LLM-based grounders quantize time into a token vocabulary and therefore trade fine boundary precision for reasoning ability. Scaling to hour-long input is a separate axis, because early fusion concatenates query and video tokens and blows up memory with , while late fusion keeps the video encoding query-independent so features can be cached and reused across thousands of queries.

Figure 2: Each pipeline stage contributes its own failure mode: the sampling stride fixes the resolution floor, the frozen per-frame encoder discards motion and ordering, early fusion makes cost quadratic in frame count, and a single-span head cannot express multiple or absent moments. The bottom timeline shows why the threshold choice dominates reported numbers: the same prediction passes at and fails at .

A region-based VLM has to turn a geometric object (a box, a point, a scribble) into something a transformer can consume, and there are only three places to put it: the token sequence, the positional encoding, or the attention logits. The cheapest route serializes the geometry as coordinate tokens, either quantized bins added to the vocabulary (Kosmos-2 uses 1024 location tokens over a 32×32 grid, written like <loc_0512>) or plain digit strings (Shikra), so the box reaches self-attention as ordinary keys and values with zero architecture change. The second route extracts region features with RoIAlign or a learned sampler and projects them into the language model’s embedding space as extra tokens interleaved into the prompt (GPT4RoI, Ferret), which every text token can attend to like a word. The third route never adds tokens at all: it adds a bias matrix to before the softmax, so patches inside the referred region get a boost and patches outside get suppressed or hard-masked to . A fourth, architecture-free trick draws the prompt in pixel space (ViP-LLaVA’s overlaid arrows and circles, Set-of-Mark’s numbered masks) and lets the vision encoder’s own self-attention pick it up, which is the only option when the model is a frozen API. The choice is a trade among spatial precision, sequence growth, and how much grounding supervision you can afford to train on.

(1) Three Injection Points: geometry enters as tokens, as positional/coordinate embeddings, or as an additive term inside the attention logits, and most systems combine at least two.
(2) Coordinate Tokens Are Free But Quantized: a 1000-bin normalized grid on a 1024 px image gives roughly 1 px resolution, but the same bins on a 4K crop after tiling lose fidelity and shift under aspect-ratio padding.
(3) Region Tokens Buy Content, Not Just Location: a pooled RoIAlign vector carries appearance of the referent, so the model can describe a region it could not have isolated from coordinates alone.
(4) Attention Bias Adds Zero Parameters: is added elementwise to the pre-softmax scores, leaving the cost and the KV cache unchanged, and a per-head can be learned.
(5) Hard Masks Destroy Context: setting outside the region removes exactly the surrounding evidence that relational questions need, and a fully masked row produces a degenerate softmax.
(6) Points Are Cheaper Supervision Than Boxes: Molmo’s pointing data shows a single pair is enough for counting and referring, and it avoids the box-regression ambiguity for deformable objects.
(7) Pixel-Space Prompts Work On Frozen Models: drawing the mark changes the image, not the architecture, but it occludes content and fails on thin or tiny structures.

Figure 1: Four ways a box reaches attention. Mechanisms (1) and (2) extend the token sequence so geometry becomes ordinary keys and values, mechanism (3) leaves the sequence untouched and edits the pre-softmax logits, and mechanism (4) modifies the pixels so the vision encoder does the work. Only (3) avoids sequence growth entirely.

The region-feature path is the one that most resembles classical detection heads: pool the backbone feature map over the box (RoIAlign → linear projector → language-model token), then concatenate a coordinate embedding so the model knows where the appearance came from, not only what it looks like. Ferret generalizes this with a spatial-aware visual sampler that samples and pools features inside an arbitrary free-form shape, so a point, a box, and a scribble all reduce to the same fixed-size token, and Groma pushes localization into the tokenizer itself by proposing regions up front and giving each one a referable ID token. The attention-bias path is complementary and is the literal reading of the question: because the softmax is invariant to nothing but its own inputs, a single additive term reweights how much probability mass a text query spends on patches inside versus outside the referent, and a moderate concentrates mass without deleting surrounding context. In practice, training matters more than the plumbing: none of these mechanisms produces reliable grounding without a large corpus of region-text pairs, which is why grounded VLMs are trained on visual genome, RefCOCO-style referring expressions, and synthetically generated region captions.

Mathematical Formulation:

Where:

• is the region token appended to the prompt, and is the full input sequence of text, region, and image-patch embeddings.
• is the vision-encoder feature map, the box in pixels, the image side used for normalization, and the normalized coordinate vector.
• projects the pooled region feature into the language-model width and embeds the coordinates; both are the only new parameters this path needs.
• are the query and key projections of , the per-head dimension, and the attention matrix over all positions.
• is the additive region bias, the set of positions that query is encouraged to attend to (the patches overlapping the referred box), and the per-head bias strength; recovers plain attention and gives a hard mask.
• is the number of visual plus text tokens, the number of added region or coordinate tokens, and the model width, so token-based injection is quadratic in while the bias term is free.

Figure 2: The same query, the same keys, three values of . A soft bias raises in-region mass from a small baseline to the majority while preserving outside context, which is what relational questions need; hard masking reaches 100% in-region mass and throws that context away.

The two designs differ in where the modalities meet. Late fusion (the modular VLM recipe behind BLIP-2, LLaVA, and Qwen2-VL) keeps a separately pretrained vision encoder, compresses its output through a small connector (an MLP projector, a Q-Former, or gated cross-attention), and feeds the resulting soft tokens into a pretrained text LLM, so nearly all capacity is inherited and only the connector plus a light fine-tune is trained. Early fusion (native multimodal, as in Fuyu-8B, Chameleon, and Llama 4) discretizes or linearly projects raw patches into the same token stream as text and runs one shared transformer from layer 0, so every layer attends across modalities and the model can emit image tokens as well as text. Early fusion is representationally stronger and, per recent scaling-law work, not worse per FLOP, so its problems past 100B parameters are systems and data problems rather than a representational ceiling. You cannot reuse a trained text-only 100B checkpoint, a Chinchilla-scale run now wants roughly 2T interleaved tokens that do not exist at text quality, the shared dense weights must be split between modalities, and every image inflates the sequence that attention pays for.

(1) Meeting Point Defines The Family: late fusion joins modalities after a frozen encoder has already compressed the image; early fusion joins them at the token level, before any transformer layer.
(2) Checkpoint Reuse Is The Real Asymmetry: late fusion amortizes the trillions of text tokens already spent on the LLM, while a native 100B model pays that bill again on scarcer interleaved data.
(3) Modality Competition In Shared Weights: a dense early-fusion stack allocates the same parameters to pixels and text, so raising the image-token fraction costs text-benchmark quality and turns into a hyperparameter you cannot sweep cheaply at 100B.
(4) Token Budget Dominates The Context: one 512px image is about 1024 VQ tokens, so a four-image document spends 4096 tokens on pixels and quadratic attention absorbs the difference.
(5) Optimization Instability Grows With Width: heterogeneous token statistics drive logit growth and loss spikes, which is why Chameleon needed QK-Norm, reordered normalization, and dropout to keep a 34B run stable.
(6) Modularity Versus Capability: late fusion lets you swap the encoder or raise resolution for the price of a connector re-train, while early fusion buys interleaved any-to-any generation that a frozen tower cannot express.

Figure 1: Late fusion (left) is an assembly of pretrained parts, so the trainable surface is a projector of roughly 0.02B parameters and the alignment budget is on the order of 1B image-text tokens. Early fusion (right) is one homogeneous stack from random init, which is simpler to shard but means the whole 100B must be pretrained on interleaved data.

The scaling wall past 100B is mostly arithmetic. A dense 100B model trained compute-optimally wants about 2T tokens and roughly FLOPs, and in native early fusion those tokens must be interleaved image-text documents; the supply of such data at web-text quality is far smaller than the text corpus, so teams either repeat data or dilute the text share and watch reasoning benchmarks regress. Late fusion sidesteps this entirely because the expensive part is already paid: aligning a connector on a couple of billion tokens with a frozen backbone is three orders of magnitude cheaper than a from-scratch native run at the same parameter count. On top of the data problem, dense early fusion suffers capacity contention, since gradients from visual reconstruction and from language modeling compete for the same MLP weights, and sparsity is the standard fix: a mixture-of-experts stack with modality-aware routing gives each modality its own parameters while keeping active FLOPs fixed, which is why recent native models are almost always sparse. Early fusion does win on the systems side (no separate vision tower to shard, no encoder-LLM pipeline bubble, and a single tokenizer path), and it is the only option when the target task requires interleaved generation or genuinely fine-grained grounding that a pooled 576-token projection has already destroyed.

Mathematical Formulation:

Where:

• is the generated output; the first two equations are the late-fusion path and the third is the early-fusion path, where a single stack consumes the whole interleaved sequence.
• is the image and the text; is the pretrained vision encoder, the text embedding, and the connector, typically the only trained matrix during alignment.
• for are the interleaved tokens, either VQ image codes or linear patch projections placed in the same sequence as text tokens.
• is the parameter count, the training tokens, and the training FLOPs; is the Chinchilla compute-optimal ratio, which for demands about 2T tokens.
• is the layer count and the model width, so shows that image tokens entering the shared stack are charged quadratically, not linearly.
• is the image-token fraction of the pretraining mixture, so is the surviving text budget; at fixed , every point of is taken directly out of language modeling.

Figure 2: Left: from-scratch native training scales as under the compute-optimal token rule, while connector alignment scales linearly in , so the gap at 100B is roughly three orders of magnitude. Right: the number of tokens an image consumes spans about an order of magnitude across tokenizers, and in early fusion every one of those tokens enters the shared quadratic attention rather than being pooled away first.

A unified model does not bolt encoders onto a finished language model at inference time; it is pretrained from step zero on interleaved sequences in which every modality has already been converted into tokens that live in the same -dimensional embedding space. Text passes through a SentencePiece vocabulary, images and sampled video frames through a ViT-style patch encoder, and audio through a speech encoder in the Universal Speech Model (USM) lineage operating on 16 kHz waveforms; each front end emits discrete ids or continuous soft tokens that are projected to the model width and concatenated in timestamp order into one causal sequence. From that point there is no separate fusion module: ordinary self-attention in every layer is the fusion mechanism, so a text token attends to an audio window from second 12 and to the patch tokens of the frame shown during that second at the same cost as attending to another word. The consequences of native tokenization are therefore budgetary rather than architectural, because Gemini’s published token rates are roughly 258 tokens per image tile or sampled frame and 32 tokens per second of audio, so one hour of video with its soundtrack consumes about 1M tokens and long context stops being a feature and becomes a precondition.

(1) Native Tokenization, Not An Adapter: multimodal data is present in pretraining from the first step, so the shared representation is learned jointly rather than stitched together by a projector trained on top of two frozen towers.
(2) One Shared Embedding Space: a per-modality encoder plus a linear projection maps sub-words, patches, and audio frames into the same , which is what makes concatenation legal.
(3) Self-Attention Does The Cross-Modal Work: there is no cross-attention adapter per modality, and the price is a prefill over the combined length.
(4) Timestamp-Ordered Packing: tokens from the same second of an audio-visual clip are placed adjacent, so co-occurring events are a few hundred positions apart instead of hundreds of thousands.
(5) Token Rates Set The Budget: 258 tokens per frame, 32 tokens per audio second, and 1 fps frame sampling as the lossy compression knob that decides what the model can even see.
(6) The Output Side Can Be Multimodal Too: a single next-token head over a vocabulary extended with discrete image and audio codec tokens lets one decoder emit text, pixels, or speech without a separate generator.

Figure 1: Three front ends, one sequence, one stack. The only modality-specific parameters sit in the encoders and projections; after that the tokens are indistinguishable to the transformer, and the output modality is decided purely by which token ids the single next-token head emits.

Two design decisions do most of the work. The first is early fusion: because all three streams enter the same stack as tokens, cross-modal alignment is learned by the same attention weights that learn syntax, which is what lets the model join a spoken sentence to whatever was on screen while it was said. A late-fusion alternative, where a frozen vision tower is glued to a frozen LLM through a projector or Flamingo-style cross-attention adapter, is far cheaper to train but confines modality interaction to the adapter, while a cascade (audio → ASR → LLM → TTS) discards everything the transcript does not carry, including speaker identity, prosody, laughter, and overlapping non-speech events. The second decision is packing order. Placing the frame tokens and the audio tokens of the same second next to each other keeps relative position encodings informative and keeps local attention patterns useful, whereas a modality-blocked layout pushes a frame and its own soundtrack thousands of positions apart. Frame rate is then the compression knob: 1 fps is adequate for scene-level questions but structurally unable to represent a 200 ms gesture, a single flashed frame, or the exact moment a door closes.

Figure 2: The same tokens, two layouts. Time-interleaved packing keeps each frame beside the 32 audio tokens recorded during it, while modality-blocked packing separates them by 1,032 positions, which is why audio-visual grounding degrades even though attention can technically still reach across the whole sequence. The width ratio also shows the real cost structure: vision dominates the budget roughly 8 to 1 over audio.

Mathematical Formulation:

Where:

• is the -th token embedding in the shared space and the raw unit behind it (a sub-word, an image tile, or a short audio window).
• is the front end for modality and its projection to the model width ; text is an embedding lookup over discrete ids, vision and audio produce continuous soft tokens.
• is the single causal sequence, ordered by timestamp rather than by modality, and is its length.
• is the text token count, the number of image tiles or sampled frames at 258 tokens each, and the audio duration in seconds at 32 tokens each.
• The prefill term is quadratic in while the KV cache grows linearly, so audio and video inflate both compute and memory before a single output token is produced.
• is the generated output drawn from one next-token distribution whose vocabulary may include discrete image and audio codes alongside sub-words.

Token Budget For One Hour Of Video With Sound:

A single hour of ordinary video therefore saturates a 1M-token window, which explains why native multimodality and million-token context arrived together in Gemini 1.5 rather than as separate features. It also explains where production effort actually goes: not into inventing a fusion block, but into deciding frame rate, tile count, and audio span so that the useful evidence survives tokenization, and into paying the quadratic prefill only for the segments that matter.

Figure 3: Native tokens are not cheap. At published rates, a spoken transcript costs about 200 tokens per minute while the same minute of video plus audio costs roughly 17,400, so the entire 1M-token context is spent after about 57 minutes. Every design choice about frame rate, tiling, or clip trimming is a move along this line.

An interleaved vision-language architecture flattens a document into one sequence that alternates text spans and images, projects each image’s patch features into the language model’s embedding space, and trains a single decoder with ordinary next-token prediction over the mixed stream. This is early fusion: visual tokens occupy real sequence positions, so they are keys and values for every text token in every layer, and text tokens are keys and values for the image. The alternative family, cross-attention conditioning (Flamingo, Idefics1), keeps vision outside the text sequence: a frozen ViT plus a Perceiver resampler compresses each image to a fixed latents, and newly inserted gated cross-attention layers let the frozen LM read those latents without ever lengthening its own sequence. The consequences are structural rather than cosmetic. Early fusion makes cost quadratic in the joint length and demands full multimodal pre-training, but it gives uniform bidirectional-within-causal mixing, unlimited image ordering, and (with discrete VQ image tokens) the ability to generate images from the same head. Cross-attention decouples image count from the text sequence and protects a strong frozen LLM, at the price of a hand-designed conditioning path, per-image attention masks that weaken multi-image relational reasoning, and no route to image output.

(1) Where Fusion Happens: early fusion concatenates projected patch embeddings into the token stream, so mixing occurs in every self-attention layer; cross-attention confines mixing to a handful of inserted layers reading a fixed latent set.
(2) Sequence Length Is The Cost Model: a ViT-L/14 at 336 px yields tokens per image, so eight images add 4608 positions and roughly the attention cost of a text-only forward pass.
(3) Attention Pattern And Multi-Image Reasoning: Flamingo masks each text token to cross-attend only to the most recent preceding image, while early fusion lets any token attend to all earlier images and their surrounding text.
(4) Parameter Budget And Text Regression: cross-attention adds new trainable blocks around a frozen LM and preserves text-only quality by construction; early fusion updates the whole backbone and risks catastrophic forgetting unless text data is replayed.
(5) Generation Symmetry: only the interleaved token-in-sequence form can emit images, because image outputs must live in the same vocabulary the head predicts (Chameleon, Emu3).
(6) Data Requirement: both families need interleaved web corpora such as M3W or OBELICS rather than caption pairs, since few-shot in-context learning is what interleaving buys.

Figure 1: The two conditioning paths. In early fusion the image becomes sequence positions inside the shared stack, so every layer mixes modalities and the sequence grows with each image. In cross-attention conditioning the image stays outside the text sequence as a fixed set of latents, read through gated layers whose gate starts at zero so the frozen LM is initially unchanged.

Practice has converged on early fusion, and the reason is mostly negative: the cross-attention path is an extra design surface with its own hyperparameters (how many layers to insert, resampler depth, latent count, gate schedule) and it caps the visual information at latents regardless of image resolution, which destroys dense OCR and chart reading. Ablations in MM1 found the interleaved token-in-sequence recipe matched or beat cross-attention at equal scale while being simpler, and the lineage of the same lab’s open models makes the shift explicit: Idefics1 (Flamingo-style cross-attention) → Idefics2 (early fusion with pooled visual tokens). Early fusion then imports the cross-attention family’s one real advantage, token compression, without its plumbing: Idefics3 pixel-shuffles patch blocks into one token, Qwen2-VL merges adjacent patches in the projector, and both keep the compressed tokens in the main sequence. What remains genuinely hard in early fusion is positional encoding for a 2D object embedded in a 1D stream, handled by explicit row-separator tokens in Fuyu or by multimodal rotary embeddings that give an image token separate height, width, and time indices.

Mathematical Formulation:

Where:

• are the image height and width and the patch size, so is the visual tokens per image: at this is .
• is the number of text tokens, the number of images in the document, and the joint sequence length the early-fusion decoder actually processes; is the model width.
• is the self-attention cost, quadratic in , so each added image contributes both its own block and cross terms with all existing tokens.
• is the raw image, the resampled latents, and the latent count ( in Flamingo), which is why is linear in and leaves the LM’s own untouched.
• are the frozen LM hidden states at layer and a learned scalar gate initialized to zero, so makes the inserted block an identity at step 0 and training degrades the language model gradually rather than abruptly.

Figure 2: Why token count dominates the choice. Early fusion pays in the joint sequence, so eight full-resolution images cost roughly a text-only pass, while cross-attention stays near because its cost is linear in . Patch pooling recovers most of the gap without leaving the sequence, which is the modern compromise.

All three designs answer one question: where does the frozen vision encoder’s patch grid meet the language model? Linear/MLP projection (LLaVA, LLaVA-1.5, Qwen2-VL, InternVL) maps every patch embedding into the LLM’s token space with a two-layer MLP and concatenates the result onto the text sequence, so vision and language share one self-attention stack. Perceiver Resampler / Q-Former (Flamingo, BLIP-2) inserts a small cross-attention module in which a fixed set of learned queries (64 latents in Flamingo, 32 in BLIP-2) attends over the patch tokens and emits exactly vectors, decoupling the LLM’s sequence length from image resolution. Gated cross-attention fusion (Flamingo’s xattn-dense blocks, Llama 3’s vision adapter, NVLM-X) never puts image tokens in the LLM sequence at all: new cross-attention layers are interleaved into the language stack (every fourth layer in Llama 3) and read the patch tokens as an external key/value memory, with a tanh gate initialized to zero so the model starts out behaviorally identical to the text-only LLM. The trade-off is compute versus fidelity versus intrusiveness: MLP concatenation is the simplest and preserves the most visual detail but makes prefill grow as ; resampling is cheapest and constant-cost but imposes a hard information bottleneck; cross-attention keeps text throughput almost untouched and protects a frozen LLM, at the price of new parameters and a more complex training recipe.

(1) Where The Visual Tokens Live: MLP projection puts them inside the LLM sequence, a resampler puts a compressed of them inside, and cross-attention keeps them outside as an external memory.
(2) Prefill Cost Scaling: concatenation pays per layer, a resampler pays with , and cross-attention pays only in the inserted layers.
(3) Added Parameters: LLaVA-1.5’s connector is roughly 20M parameters of MLP, BLIP-2’s Q-Former is about 188M with a BERT-base initialization, and Llama 3’s cross-attention adapter adds tens of billions at the 405B scale.
(4) Information Bottleneck: a fixed caps how much of a high-resolution image can survive, which is why resampler-based models underperform on dense OCR, charts, and counting.
(5) Frozen Versus Tuned Backbones: the Idefics2 ablation found cross-attention wins when the LLM is frozen, while the fully autoregressive concatenation design wins once the LLM is unfrozen.
(6) Token Count Beats Connector Design: MM1’s ablations show the connector type matters far less than image resolution and visual token count, which is why the field converged on MLP plus cheap token compression.

Figure 1: The three fusion families differ only in where the patch tokens are allowed to go. Concatenation admits all of them into the shared sequence, a resampler admits a fixed , and gated cross-attention admits none, reading them instead as an external memory through layers whose tanh gate starts at zero.

The training recipe follows from the architecture. An MLP connector is so small that a two-stage schedule with about 558K caption pairs for alignment and 665K instruction samples for tuning is enough to reach state-of-the-art benchmark scores, which is what made LLaVA-1.5 reproducible on a single node. A Q-Former is a real transformer that must learn what to query, so BLIP-2 needs a dedicated representation-learning stage (contrastive, matching, and captioning objectives) before the generative stage, and remains the least data-efficient of the three per unit of final accuracy. Cross-attention sits in between: the gate makes optimization stable and lets you keep the language model frozen, so text benchmarks cannot regress, but you are training new layers that must learn to be useful without ever seeing visual tokens in the residual stream. NVLM’s controlled comparison is the cleanest evidence on the trade-off, finding the decoder-only concatenation variant stronger on OCR and multimodal reasoning while the cross-attention variant gave better throughput on high-resolution inputs.

Mathematical Formulation:

Where:

• is the input image, the vision encoder (typically a frozen CLIP or SigLIP ViT), and its patch token grid.
• is the projected visual sequence, with the connector weights and a GELU nonlinearity; a single recovers the original linear projection of LLaVA-1.
• is the patch count ( for ViT-L/14 at 336 px), the text length, the LLM sequence length, the LLM width, and the vision width.
• are the learned latent queries and the resampled output; is fixed at design time, so has the same size for a thumbnail and for a 4K page.
• is the text hidden state inside the LLM and the scalar gate parameter, initialized so that and the pretrained language behavior is exactly preserved at step zero.
• is the per-layer self-attention cost that concatenation inflates, and the per-layer cost of a cross-attention block, which is linear in rather than quadratic.

Figure 2: Prefill attention cost relative to a text-only forward pass at . At visual tokens, concatenation costs about the text-only baseline because the whole stack pays , gated cross-attention costs about since only 8 of 32 layers see the image and they scale as , and a 64-latent resampler stays near because the LLM never sees more than positions.