DL0145 ITC, ITM, and MLM Pretraining Losses

The three objectives differ along three axes: how much cross-modal interaction they permit, which parameters receive gradient, and what the pretrained model can actually do afterwards. ITC is a dual-encoder loss that compresses each image and each caption into a single L2-normalized vector and maximizes the cosine similarity of matched pairs against the other pairs in the batch, so it produces embeddings that can be precomputed and indexed, at the price of never letting a word look at a patch. ITM is a cross-encoder loss: image and text tokens are fused with cross-attention and a binary head decides match or no-match, which gives token-level discrimination but requires one joint forward pass per candidate pair, so it can only ever be a reranker. MLM masks about 15% of the text tokens and reconstructs them conditioned on the image, which is the objective that forces word-to-region grounding and trains the language head that downstream VQA and captioning heads reuse. CLIP trains ITC alone at batch 32,768; ALBEF and BLIP train all three at equal weight because each one supplies something the others cannot.

(1) ITC Learns Global Alignment: an InfoNCE objective over the in-batch similarity matrix, with negatives per anchor and a learned temperature, optimized symmetrically image-to-text and text-to-image.
(2) ITM Learns Fine-Grained Verification: binary cross-entropy on the fused state, and it is nearly useless unless the negatives are hard negatives mined from the ITC similarity distribution.
(3) MLM Learns Grounded Language: reconstructing masked words from surrounding text plus image evidence ties nouns and attributes to specific patches and yields a vocabulary-sized output head that pure ITC models do not have.
(4) Different Architectural Footprint: ITC updates only the two unimodal towers and their projections, while ITM and MLM update the cross-attention fusion layers; only ITC leaves behind vectors you can put in an ANN index.
(5) Different Compute Profile: ITC is one forward pass per modality but needs a huge batch or a momentum queue (65,536 in ALBEF), whereas ITM and MLM add fusion passes, so BLIP runs its text stack three times per step.
(6) They Compose Into A Pipeline: ITC → hard-negative mining → ITM rerank is the standard retrieval stack, and MLM is the branch that makes the same weights usable for generation-flavored tasks.

Figure 1: The same two towers, three different graphs. ITC stops at a single pooled vector per modality, so no gradient ever reaches a fusion layer; ITM and MLM share the cross-attention stack and differ only in the head placed on top, which is why adding MLM to an ITM model is cheap while adding either to a CLIP-style model requires new parameters.

The three losses are not interchangeable because they fail in different directions. An ITC-only model scores “a man in a red shirt riding a bike” and “a man in a blue shirt riding a bike” almost identically, since a single 512-dimensional vector cannot preserve every attribute binding; this is the classic bag-of-words behavior that compositionality benchmarks expose. ITM fixes exactly that case because the fused representation can compare the color word against the actual patches, but its cost is joint forward passes for a gallery of images and captions, which is why ALBEF and BLIP use ITC to shortlist candidates and only then run ITM. MLM contributes something neither of the other two does: because the target is a token id rather than a pair label, it forces the model to represent which region licenses which word, and it produces the token-level head that VQA fine-tuning reuses. The coupling runs the other way too, since the ITM negative sampler reads its distribution from the ITC matrix, so a weak ITC head starves ITM of informative negatives.

Figure 2: One batch, two uses of one matrix. ITC consumes the whole row and the whole column as a softmax classification problem, while ITM samples a single off-diagonal entry with high similarity and feeds that pair to the fusion encoder labelled no-match. The six near-zero bars are the reason random negatives teach ITM almost nothing: the easy cases already carry negligible gradient.

Mathematical Formulation:

Where:

• and are the L2-normalized projections of the image and caption taken from the two unimodal towers, so is a cosine similarity in .
• is the learned temperature (CLIP initializes it at 0.07 and clamps it), which controls how sharply the softmax concentrates on the hardest negatives.
• index the batch, and the diagonal holds the positives; is the same expression with the softmax taken down the column.
• is the fused state after cross-attention and the match label, with the negative pair drawn from the ITC row rather than uniformly.
• is the set of masked positions (about 15% of tokens), the unmasked context, and the true word piece at position out of roughly 30k classes.
• The total loss in ALBEF and BLIP uses equal weights, and MLM replaces the plain text-encoder forward pass rather than adding an independent one.

Retrieval Cost For 1k Images And 5k Captions:

ITC needs 6,000 encoder passes total and then only dot products, which is what makes billion-scale ANN search possible. A pure cross-encoder needs 5,000,000 fusion passes on the same Flickr30k-sized benchmark, while reranking the top ITC candidates needs 160,000, about 31 times cheaper than the exhaustive version and still recovering most of the accuracy gain. That arithmetic, not any statement about representational quality, is why production retrieval systems keep ITC in the serving path and treat ITM as an offline or second-stage model.