DL0098 CLIP Zero-Shot Classification

Written by

What is CLIP, and how does contrastive image-text training enable zero-shot classification?

Answer

CLIP (Contrastive Language-Image Pre-training) is a dual-encoder model: an image encoder (a ViT or ResNet) and a text encoder (a Transformer) each map their input to a vector in one shared embedding space, and the vectors are L2-normalized so their inner product is a cosine similarity. Training uses a batch of $N$ image-caption pairs scraped from the web (400M pairs for the original model), computes the full $N \times N$ similarity matrix, and applies a symmetric InfoNCE loss that pushes the $N$ diagonal entries up and the $N^2 - N$ off-diagonal entries down. Nothing in that objective mentions class labels, so the learned text encoder is a general classifier generator: at test time you write each candidate class as a sentence such as “a photo of a golden retriever”, encode it, and the resulting unit vectors act as the rows of a linear classifier weight matrix. Classification is then one matrix multiply followed by an argmax over cosine similarities, which is why a new label set costs a text-encoder forward pass rather than a training run. The best original model (ViT-L/14@336px) reaches 76.2% zero-shot top-1 on ImageNet, matching a supervised ResNet-101 with an identical 76.2% ImageNet top-1 that saw 1.28M labeled images, and it is far more robust under distribution shift.

(1) Two Encoders, One Space: each modality has its own encoder plus a linear projection into a shared $d$ -dimensional space; there is no cross-attention between them, so image and text embeddings can be computed and cached independently.
(2) Symmetric InfoNCE Over The Batch: the loss is cross-entropy applied twice, once along rows (image picks its caption) and once along columns (caption picks its image), averaged.
(3) Learned Temperature: the logit scale $\exp(\tau)$ is a trained scalar initialized to a temperature of 0.07 and clipped at 100, which controls how hard the softmax pushes against near-miss negatives.
(4) Prompts Become Classifier Weights: zero-shot inference replaces the learned classification head with text embeddings of class descriptions, so the label set is defined at inference time, not at training time.
(5) Batch Size Is Part Of The Objective: negatives come only from the current batch, so the original model trained at $N = 32768$ across many GPUs with the similarity matrix sharded.

Diagram of CLIP training: a batch of N images passes through an image encoder to unit-norm vectors, a batch of N captions passes through a text encoder to unit-norm vectors, and both feed an N by N cosine similarity matrix whose diagonal cells are the positive pairs

Figure 1: Each training step builds the full $N \times N$ similarity matrix; the $N$ diagonal entries are the true pairs and every other entry is an in-batch negative, which is why one step at $N = 32768$ supplies over a billion contrastive comparisons.

Three details separate a working CLIP from a broken one. Normalization is load-bearing: without it the model can shrink the loss by inflating embedding norms instead of improving alignment, and the learned temperature then has no fixed scale to calibrate against. Prompt wording matters more than people expect, because captions in the training data are sentences, not bare nouns: using “a photo of a {label}” instead of the raw class name adds about 1.3 points on ImageNet, and ensembling 80 prompt templates by averaging their normalized text embeddings adds roughly 3.5 points for almost no inference cost, since the averaged vectors are computed once and cached. Finally, label naming is part of the model: polysemous class names such as “boxer” (dog breed or athlete) or “crane” (bird or machine) must be disambiguated in the prompt, which is a form of engineering that has no analogue in a supervised classifier with integer labels.

Mathematical Formulation:
$z_i = f_{\theta}(I_i) / \|f_{\theta}(I_i)\|_2$
$t_j = g_{\phi}(T_j) / \|g_{\phi}(T_j)\|_2$
$s_{ij} = \exp(\tau)\, z_i^{\top} t_j$
$\mathcal{L}_{I} = -\frac{1}{N}\sum_{i} \log \frac{e^{s_{ii}}}{\sum_{j} e^{s_{ij}}}$
$\mathcal{L} = \frac{1}{2}\left(\mathcal{L}_{I} + \mathcal{L}_{T}\right)$
$\hat{y} = \arg\max_{c} \; z^{\top} t_c$

Where:

$z_i$ and $t_j$ are the unit-norm embeddings of image $I_i$ and caption $T_j$ , both in $\mathbb{R}^{d}$ with $d = 768$ for ViT-L/14.
$f_{\theta}$ is the image encoder plus its projection and $g_{\phi}$ the text encoder plus its projection; the two share no weights.
$s_{ij}$ is the scaled cosine similarity and $\exp(\tau)$ the learned logit scale; since $z_i^{\top} t_j$ lies in $[-1, 1]$ , the scale alone sets the sharpness of the softmax.
$i, j \in \{1, \ldots, N\}$ index the batch; $\mathcal{L}_{I}$ normalizes each row and $\mathcal{L}_{T}$ the same expression with the softmax taken over columns.
$\hat{y}$ is the zero-shot prediction for a query image with embedding $z$ , and $t_c$ is the (optionally prompt-ensembled and renormalized) text embedding of class $c$ ; the temperature drops out of the argmax.
Required initial condition: $\tau = \log(1/0.07)$ at step 0, clamped so that $\exp(\tau) \leq 100$ throughout training to prevent the logits from exploding.

Grouped bar chart comparing top-1 accuracy of a supervised ResNet-101 and zero-shot CLIP ViT-L/14 on ImageNet and five distribution-shift benchmarks, with CLIP far ahead on ImageNet-R, ObjectNet, Sketch and ImageNet-A

Figure 2: Matched at 76.2% on the ImageNet validation set, the two models diverge sharply under shift: zero-shot CLIP keeps 77.1% on ImageNet-A and 60.2% on Sketch, where the supervised model drops to 2.7% and 25.2%, evidence that never fitting the ImageNet label distribution is itself a robustness mechanism.

Property	Zero-shot CLIP (ViT-L/14@336px)	Supervised classifier (ResNet-101)
Task supervision	Zero labeled examples for the task, 400M noisy web pairs for pretraining	1.28M hand-labeled images for exactly this label set
Adding a class	Encode one more prompt and append the vector, no gradient step	Collect labels, then retrain or refit the head
ImageNet top-1	76.2%	76.2%
ImageNet-A top-1	77.1%	2.7%
Specialized domains	Weak: near chance on tumor-patch classification, poor on counting and satellite land use	Strong once in-domain labels exist
Inference cost	One image forward pass plus a $d \times C$ matmul against cached text vectors	One forward pass through a fixed head

GenAI Transformer

DL0098 CLIP Zero-Shot Classification

Comments

Leave a Reply Cancel reply

More posts

MSD0064 Notification Ranking System

MSD0063 Image Copyright Detection

MSD0062 Ads Ranking System

MSD0061 Nearby Place Recommendation