DL0099 Masked Autoencoding MAE

Masked autoencoding is a denoising self-supervised objective: hide a random subset of the input units, then train the network to reconstruct exactly what was hidden from the surviving context. The supervision is free because the labels are the withheld input itself, and the representation quality comes from the fact that reconstruction cannot be solved by copying: predicting a masked word or a masked image region requires the encoder to carry semantic, long-range context in its hidden states. BERT-style masked language modeling is the discrete version, masking about 15% of tokens and predicting a softmax over the vocabulary at those positions. MAE is the continuous vision version, masking about 75% of image patches and regressing raw pixels with an asymmetric encoder-decoder: a deep ViT encoder that only ever sees the visible 25% of patches, plus a shallow decoder that is discarded after pretraining. That asymmetry is what makes it cheap, and it scales: a ViT-Huge pretrained with MAE on ImageNet-1K alone reaches 87.8% top-1 after fine-tuning, with over 3x faster pretraining than a full-token baseline.

(1) Masking Manufactures The Labels: no annotation is required, so the objective scales with raw data; the only design decisions are what to mask, how much, and what target to regress or classify.
(2) Asymmetric Encoder-Decoder: the encoder processes only unmasked tokens and mask tokens enter at the decoder, so both compute and the pretrain-finetune input mismatch shrink dramatically.
(3) Masking Ratio Tracks Information Density: text is dense and semantic so 15% suffices, while adjacent pixels are highly redundant and need 75% before the task stops being local interpolation.
(4) The Target Defines The Representation: raw pixels, per-patch normalized pixels, discrete VQ tokens (BEiT), or latent teacher features (data2vec) all train the same encoder toward different levels of abstraction.
(5) Strong Fine-Tuning, Weak Linear Probing: MAE features are highly non-linear, so they beat contrastive methods after fine-tuning but lag them under a frozen linear probe or k-NN retrieval.

Figure 1: The MAE pipeline for a ViT-L/16 at 224 pixels: of 196 patches only 49 reach the encoder, mask tokens are inserted just before a shallow decoder, and the MSE loss is computed on masked patches only.

Two details separate the text and vision instantiations. BERT cannot drop masked positions because the prediction head must sit at those positions, so the literal [MASK] symbol is fed to the encoder during pretraining and never appears at fine-tuning time; the original recipe patches this discrepancy by replacing only 80% of selected tokens with [MASK], 10% with a random token, and leaving 10% unchanged. MAE removes the problem structurally: masked patches are simply deleted from the encoder’s input sequence, and the mask token is a learned vector injected at the decoder. The second detail is the loss support. Computing the reconstruction error on masked patches only matters, since including visible patches lets the model spend capacity on an identity mapping and measurably hurts downstream accuracy.

Mathematical Formulation:

Where:

• is the cross-entropy over masked token positions and the per-patch MSE; both are averaged over masked positions only.
• is the true content at position , the prediction, and the corrupted sequence containing symbols.
• and are the masked and visible index sets, disjoint with for total tokens or patches.
• is the deep encoder applied to visible tokens alone, the shallow decoder, and the shared learned mask-token embedding plus positional encoding.
• is the masking ratio: for BERT and for MAE, giving of patches.
• is quadratic self-attention cost, so at encoder attention drops to and the token-wise MLP cost to ; targets are usually per-patch normalized pixels rather than raw values.

Figure 2: Approximate ViT-L behaviour versus masking ratio: fine-tuning is nearly flat from 40% to 80%, while linear probing peaks sharply near 75%, which is why a ratio that looks extreme for text is the vision default.