DL0136 Unified Multimodal Gemini Architecture

A unified model does not bolt encoders onto a finished language model at inference time; it is pretrained from step zero on interleaved sequences in which every modality has already been converted into tokens that live in the same -dimensional embedding space. Text passes through a SentencePiece vocabulary, images and sampled video frames through a ViT-style patch encoder, and audio through a speech encoder in the Universal Speech Model (USM) lineage operating on 16 kHz waveforms; each front end emits discrete ids or continuous soft tokens that are projected to the model width and concatenated in timestamp order into one causal sequence. From that point there is no separate fusion module: ordinary self-attention in every layer is the fusion mechanism, so a text token attends to an audio window from second 12 and to the patch tokens of the frame shown during that second at the same cost as attending to another word. The consequences of native tokenization are therefore budgetary rather than architectural, because Gemini’s published token rates are roughly 258 tokens per image tile or sampled frame and 32 tokens per second of audio, so one hour of video with its soundtrack consumes about 1M tokens and long context stops being a feature and becomes a precondition.

(1) Native Tokenization, Not An Adapter: multimodal data is present in pretraining from the first step, so the shared representation is learned jointly rather than stitched together by a projector trained on top of two frozen towers.
(2) One Shared Embedding Space: a per-modality encoder plus a linear projection maps sub-words, patches, and audio frames into the same , which is what makes concatenation legal.
(3) Self-Attention Does The Cross-Modal Work: there is no cross-attention adapter per modality, and the price is a prefill over the combined length.
(4) Timestamp-Ordered Packing: tokens from the same second of an audio-visual clip are placed adjacent, so co-occurring events are a few hundred positions apart instead of hundreds of thousands.
(5) Token Rates Set The Budget: 258 tokens per frame, 32 tokens per audio second, and 1 fps frame sampling as the lossy compression knob that decides what the model can even see.
(6) The Output Side Can Be Multimodal Too: a single next-token head over a vocabulary extended with discrete image and audio codec tokens lets one decoder emit text, pixels, or speech without a separate generator.

Figure 1: Three front ends, one sequence, one stack. The only modality-specific parameters sit in the encoders and projections; after that the tokens are indistinguishable to the transformer, and the output modality is decided purely by which token ids the single next-token head emits.

Two design decisions do most of the work. The first is early fusion: because all three streams enter the same stack as tokens, cross-modal alignment is learned by the same attention weights that learn syntax, which is what lets the model join a spoken sentence to whatever was on screen while it was said. A late-fusion alternative, where a frozen vision tower is glued to a frozen LLM through a projector or Flamingo-style cross-attention adapter, is far cheaper to train but confines modality interaction to the adapter, while a cascade (audio → ASR → LLM → TTS) discards everything the transcript does not carry, including speaker identity, prosody, laughter, and overlapping non-speech events. The second decision is packing order. Placing the frame tokens and the audio tokens of the same second next to each other keeps relative position encodings informative and keeps local attention patterns useful, whereas a modality-blocked layout pushes a frame and its own soundtrack thousands of positions apart. Frame rate is then the compression knob: 1 fps is adequate for scene-level questions but structurally unable to represent a 200 ms gesture, a single flashed frame, or the exact moment a door closes.

Figure 2: The same tokens, two layouts. Time-interleaved packing keeps each frame beside the 32 audio tokens recorded during it, while modality-blocked packing separates them by 1,032 positions, which is why audio-visual grounding degrades even though attention can technically still reach across the whole sequence. The width ratio also shows the real cost structure: vision dominates the budget roughly 8 to 1 over audio.

Mathematical Formulation:

Where:

• is the -th token embedding in the shared space and the raw unit behind it (a sub-word, an image tile, or a short audio window).
• is the front end for modality and its projection to the model width ; text is an embedding lookup over discrete ids, vision and audio produce continuous soft tokens.
• is the single causal sequence, ordered by timestamp rather than by modality, and is its length.
• is the text token count, the number of image tiles or sampled frames at 258 tokens each, and the audio duration in seconds at 32 tokens each.
• The prefill term is quadratic in while the KV cache grows linearly, so audio and video inflate both compute and memory before a single output token is produced.
• is the generated output drawn from one next-token distribution whose vocabulary may include discrete image and audio codes alongside sub-words.

Token Budget For One Hour Of Video With Sound:

A single hour of ordinary video therefore saturates a 1M-token window, which explains why native multimodality and million-token context arrived together in Gemini 1.5 rather than as separate features. It also explains where production effort actually goes: not into inventing a fusion block, but into deciding frame rate, tile count, and audio span so that the useful evidence survives tokenization, and into paying the quadratic prefill only for the segments that matter.

Figure 3: Native tokens are not cheap. At published rates, a spoken transcript costs about 200 tokens per minute while the same minute of video plus audio costs roughly 17,400, so the entire 1M-token context is spent after about 57 minutes. Every design choice about frame rate, tiling, or clip trimming is a move along this line.