DL0111 SigLIP vs CLIP

CLIP and SigLIP share the same dual-encoder architecture, the same cosine-similarity scoring, and the same zero-shot classification recipe; the substantive difference is the training objective. CLIP optimizes a softmax InfoNCE loss in which each image’s logit row is normalized over every text in the batch and each text’s column over every image, so the loss contribution of a single pair depends on the entire batch, and the distributed implementation needs an all-gather plus a global normalization pass. SigLIP replaces this with a pairwise sigmoid loss that treats each of the cells of the similarity matrix as an independent binary decision (“is this a real pair?”), using a shared learnable temperature plus a learnable bias initialized to to absorb the fact that only of the pairs are positive. Because nothing is normalized across the batch, the loss decomposes into blocks, which lets SigLIP be implemented as a memory-efficient ring over devices and trained at batch sizes a softmax implementation cannot fit. Empirically the sigmoid loss is clearly better at small and moderate batch sizes, the two objectives converge above roughly 32k examples per step, and the million-example batch study SigLIP made possible showed that accuracy saturates long before that. That efficiency, plus the shape-optimized SoViT-400m/14 backbone released with it, is why SigLIP checkpoints became the default vision tower in many recent vision-language models such as PaliGemma.

(1) Loss Form: CLIP uses two symmetric softmax cross-entropies over the batch; SigLIP uses one binary logistic loss per pair, summed over the full grid.
(2) Batch Coupling: softmax logits are only meaningful after a global normalization, so every device must see all embeddings; sigmoid logits are independent, so blocks can be scored and reduced locally.
(3) Extra Bias Parameter: the sigmoid objective faces an extreme negatives against positives imbalance, which the learnable bias fixes by starting every logit near “not a pair”.
(4) Batch-Size Behavior: sigmoid wins at 4k to 16k, matches softmax above roughly 32k, and both saturate rather than improving toward one million.
(5) Everything Else Is Unchanged: image tower, text tower, unit-norm embeddings, and prompt-based zero-shot evaluation are identical, so SigLIP is effectively a drop-in replacement at the loss layer.
(6) Practical Differences In The Checkpoints: public English SigLIP models use a 32k SentencePiece vocabulary with a 64-token text context (the original paper’s initial experiments used 16, but the released checkpoints increased it to 64), shorter than CLIP’s 77-token BPE context, which matters when captions are long.

Figure 1: Both objectives see the same similarity matrix. CLIP normalizes each row and column, so a single logit’s gradient depends on the whole batch; SigLIP scores each cell as an independent binary label with a shared temperature and bias, which removes the coupling entirely.

Mathematical Formulation:

Where:

• and are the unit-norm image and text embeddings, so is a cosine similarity in .
• is the scaled logit and is the learnable temperature, parameterized in log space in both methods.
• index the global batch; the diagonal holds the true pairs and the off-diagonal holds negatives.
• for a matched pair and otherwise, which is the binary label the sigmoid loss regresses on.
• is the learnable bias unique to SigLIP, with the required initialization and ; CLIP has no analogue because its softmax is shift-invariant.

The engineering consequence is the part interviewers usually probe. In a softmax implementation the batch is a single indivisible unit: after the all-gather each of the devices holds all embeddings and materializes an logit slab, and the row and column sums must be reduced across devices before any gradient exists. SigLIP instead keeps each device’s image chunk local and passes text chunks around a ring, accumulating loss from one block at a time, so peak logit memory drops by a factor of and no cross-device normalization is needed. This is exactly what allowed the authors to sweep batch size up to one million and demonstrate saturation, and it is also why the SigLiT variant, which locks a pretrained image tower and trains only the text side, reached 84.5% ImageNet zero-shot accuracy in two days on four TPUv4 chips. The trade-off is two extra hyperparameters to get right: a badly initialized bias makes the first thousands of steps a wasted fight against the negative-pair prior.

Figure 2: The softmax loss forces an all-gather and a global normalization, so each device carries an logit slab. The sigmoid loss decomposes, so a ring of steps covers every pair while only logits exist at once.