ML0078 Self-Supervised Learning

Self-supervised learning manufactures its own labels from the structure of the data: hide part of the input and train the model to predict it. Masked words in a sentence, the next token, a rotated image’s angle, or two augmented views of the same photo all give a supervised training signal without any human annotation. The result is a pretrained representation that transfers to downstream tasks with small labeled sets. Unsupervised learning (clustering, density estimation) discovers structure with no prediction target at all, and its output is the structure itself. Self-supervised learning is supervised machinery run on invented labels; that is why it scales with the same architectures and optimizers as supervised learning.

(1) Pretext Tasks: the label is a withheld piece of the input itself, for example masked-token prediction in BERT, next-token prediction in GPT-style models, or augmentation agreement in SimCLR and SwAV.
(2) The Key Difference: unsupervised learning has no target and answers “how is the data organized”, while self-supervised learning has a target derived from the input and answers “what representation predicts the hidden parts”; both need zero human labels.
(3) Scale Evidence: Meta’s SEER pretrained a billion-parameter network with SwAV on one billion uncurated Instagram images and reached 84.2% ImageNet top-1 after fine-tuning; with only 10% of ImageNet’s labels it still hit 77.9% top-1, and with just 1% it reached 60.5%.

Figure 1: Pretext tasks invent labels from the input itself: left, mask a token and predict it from context (BERT style); right, pull the embeddings of two augmented views of the same image together (contrastive style).

The standard production pattern is two-stage. Stage one pretrains on the cheap unlabeled corpus with a pretext objective, which is where most of the compute goes. Stage two fine-tunes the frozen or lightly thawed representation on the small labeled set for the real task, which is where the labels go. The pretext objective is not the product; it is a scaffold whose only job is to force the network to learn transferable structure such as grammar, object parts, and semantic similarity.

Figure 2: The two-stage pattern: expensive self-supervised pretraining happens once on unlabeled data; cheap fine-tuning happens per task on labeled data, which is why a 1% labeled slice can still yield strong accuracy.

Mathematical Formulation:

Where:

• is the set of masked positions, the hidden token, and the visible context (masked language modeling).
• are embeddings of two augmented views of the same input, covers all candidate views, is a similarity such as cosine, and is a temperature (contrastive InfoNCE).