DL0144 Synthetic Captions vs Alt-Text

Raw alt-text is a noisy channel: it averages roughly 10 words, frequently describes the page rather than the pixels, and is polluted by filenames, SEO keywords, stock-photo boilerplate, and product codes. Running a strong captioner over the corpus replaces that with a dense, grounded description of roughly 50 words that actually mentions attributes, counts, spatial relations, and background objects, which sharply improves text-to-image alignment and prompt following in generative models and improves retrieval in contrastive models. The cost is that the captioner can only describe what it can see and what it already knows, so recaptioning silently deletes the named entities, brands, landmarks, and long-tail vocabulary that only alt-text carries, and it stamps every sample with a single writing style, which collapses caption diversity and imports the captioner’s own hallucinations as ground truth. The practical result reported across DALL-E 3, DataComp, and Recap-DataComp-1B is that neither source wins outright: the tuned quantity is the mixing ratio, with generative text-to-image training favouring almost pure synthetic data (DALL-E 3 used a 95% blend) while contrastive CLIP-style training usually peaks at a genuine mix and degrades toward 100% synthetic.

(1) Alt-Text Is Noisy But Unbiased: it is written by humans for arbitrary purposes, so it is wrong or irrelevant often, yet its errors are not correlated with any single model’s blind spots.
(2) Synthetic Captions Raise Density: a 50-word grounded description supplies far more supervised text tokens per image than a 10-word alt string, which is what drives the gain in compositional and attribute-level alignment.
(3) Recaptioning Deletes World Knowledge: a captioner that cannot name a specific landmark, celebrity, or product writes “a tall building” and the entity vanishes from the training signal permanently.
(4) Style Collapse And Inherited Hallucination: every caption inherits one syntax template and one error distribution, so the student model learns the captioner’s biases as if they were facts.
(5) The Mixing Ratio Is The Real Knob: sample the synthetic caption with probability and the alt-text otherwise; near 1 suits text-to-image generation, intermediate suits contrastive pre-training.
(6) Context Length And One-Time Compute: CLIP’s text encoder truncates at 77 tokens, so dense captions are partially discarded, and recaptioning a billion images is a fixed preprocessing bill of order GPU-hours.

Figure 1: The two text streams come from different places. Alt-text is a property of the page, synthetic captions are a property of the pixels plus the captioner’s knowledge, and the only place the two are reconciled is the mixing sampler that draws each training pair’s caption with probability .

The mechanism behind the improvement is easy to state: the contrastive or captioning objective is unchanged, only the text marginal moves. A short alt string gives the model very few positive constraints, so many wrong images remain compatible with it, whereas a dense caption pins down attributes, counts, and relations and therefore produces a much sharper positive. That is exactly why generative text-to-image models benefit most: their failure mode is ignoring adjectives, counts, and spatial prepositions in a user prompt, and dense captions are the only supervision that ever mentions those. It is also why the gains shrink for discriminative zero-shot classification at scale: with a 1B-scale pool, alt-text’s lexical diversity and entity coverage begin to matter more than its per-sample precision, and studies on DataComp report that generated captions dominate at small and medium pool sizes while a raw-plus-synthetic mixture wins at the large scale.

Figure 2: Two views of the same trade-off. Panel (a) shows why one blend cannot serve both objectives: alignment keeps improving with while zero-shot classification turns over once entity-bearing alt-text is crowded out. Panel (b) shows the second-order problem: dense captions push the length distribution against the 77-token text-encoder limit, so part of the extra supervision is truncated before it is ever used.

Mathematical Formulation:

Where:

• is the synthetic caption sampled from the captioner conditioned on image , and is the raw alt-text scraped alongside that image.
• is the caption actually used for example , and is the mixing ratio, the single hyper-parameter that decides how much of the corpus the captioner rewrites.
• and are the image and text towers producing normalized embeddings, is the learned temperature, and is the batch size supplying the in-batch negatives.
• Nothing in changes when you recaption; the entire effect flows through the conditional distribution of given , which becomes lower-noise but also lower-entropy and model-biased.

One-Time Recaptioning Cost For 1B Images:

At a sustained 20 images per second per GPU for a 7B-class captioner emitting about 50 tokens, one billion images take seconds of single-GPU time, roughly 14,000 GPU-hours, or about half a day on a 1,000-GPU cluster. That is a real but one-time preprocessing cost, amortized over every subsequent training run on the corpus, which is why recaptioning is usually cheaper than the ablation sweeps it replaces. The recurring costs are subtler: longer captions mean more text-encoder tokens per step, and a frozen captioner freezes a snapshot of one model’s competence into the dataset.