DL0126 Contrastive Learning Pairs

Contrastive learning trains an encoder by comparison instead of by prediction of a label: it pulls the embeddings of things that should mean the same thing together and pushes everything else apart. A positive pair is two views of the same underlying content (two augmentations of one photo in SimCLR, an image and its caption in CLIP, two nearby audio segments in CPC), and a negative pair is an anchor paired with content assumed to be different (any other item in the batch). The standard objective, InfoNCE, turns this into a softmax classification problem: given an anchor, identify its single positive among one positive and negatives, using scaled cosine similarity as the logit. Positives define what the representation should be invariant to, and negatives are what prevents the trivial solution where the encoder maps every input to the same vector, a failure called representational collapse. The whole design problem of a contrastive system is therefore the pair-construction policy: a positive that is too easy teaches nothing, and a negative that is secretly a positive actively teaches the wrong thing.

(1) Positives Encode The Invariance You Want: whatever transformation you apply to build a positive pair is exactly the factor the encoder learns to discard, so color jitter buys color invariance and destroys any task where color is the signal.
(2) Negatives Are The Anti-Collapse Term: without a repulsive term the constant map is a global minimum of the attraction loss, so negatives provide the uniformity pressure that spreads embeddings over the hypersphere.
(3) InfoNCE Is A -Way Classification: one positive logit in the numerator, the positive plus all negatives in the denominator, which is why the loss is bounded below by under a random encoder.
(4) Temperature Sets Hard-Negative Focus: small (0.05 to 0.1 is typical) concentrates almost all repulsive gradient on the few most similar negatives, while large treats all negatives nearly equally.
(5) Negative Count Is A Systems Problem: SimCLR needs batch sizes of 4096 or more to get enough in-batch negatives, MoCo decouples them with a momentum-encoder queue, and CLIP trained with a 32768 global batch across many GPUs.
(6) False Negatives Are The Main Bias: in-batch negatives assume every other sample is semantically different, which is false on class-imbalanced or deduplicated-poorly data, and the loss then pushes apart items that should be close.

Pairs do not have to come from augmentation. The general recipe is: find a cheap source of known agreement and treat everything else as disagreement. Augmentation gives agreement between two crops of one image; multimodal alignment gives agreement between an image and the alt-text scraped alongside it; temporal or spatial context gives agreement between adjacent frames, patches, or sentences; and labels give agreement between any two samples of the same class, which is what SupCon exploits to allow many positives per anchor. The negatives are almost always just the rest of the batch, because sampling them explicitly is expensive and in-batch negatives come for free with the forward pass already computed.

Figure 1: Pair construction and its geometric effect. Each anchor has exactly one positive (the other view of the same source) and negatives (all views of the other images in the batch); on the unit hypersphere the loss is a single attractive force toward the positive balanced against repulsive forces from every negative.

Mathematical Formulation:

Where:

• is the cosine similarity used as the logit, computed on L2-normalized embeddings produced by the encoder (and, in SimCLR, a discarded projection head).
• is the per-anchor NT-Xent loss and its average over all views of an -image batch.
• indexes the anchor view, its unique positive, and ranges over every other view, so the denominator holds 1 positive and negatives.
• is the temperature; the gradient weight on negative is its softmax probability, so shrinking sharpens that distribution onto the hardest negatives.
• is the mutual information between the two views and the number of negatives, giving the standard InfoNCE lower bound: the bound saturates at , which is one reason large batches help.

Figure 2: Temperature decides which negatives matter. At the hardest 1% of negatives absorbs most of the repulsive gradient, making training an implicit hard-negative miner that is also maximally sensitive to false negatives; at the pressure is spread nearly uniformly and the embedding space stays smoother but less discriminative.