DL0143 VLM Pretraining Data Curation

Written by

Walk through the data curation pipeline for pre-training large VLMs, including web image-text filtering, synthetic re-captioning (e.g., LLaVA-1.5/1.6), and visual instruction tuning.

Answer

Pre-training a large VLM is mostly a data engineering problem, and the pipeline produces three qualitatively different corpora: a heavily filtered web corpus for breadth, a synthetically re-captioned corpus for description quality, and a small hand-assembled instruction corpus for behaviour. Stage one takes a raw crawl on the order of 10B alt-text pairs, removes NSFW, PII and duplicate URLs, drops images below roughly 200 px and captions outside a 5 to 64 token window, keeps about the top 30% by CLIP image-text cosine, and intersects that with a cluster-based balancing filter, which is how DataComp’s 12.8B CommonPool collapses to the ~1.4B pairs of DataComp-1B. Stage two attacks the fact that surviving alt-text is still short, keyword-shaped and frequently describes the page rather than the pixels: a captioner VLM, itself trained on a small set of 100K high-quality dense captions in the ShareGPT4V style, rewrites each image into a 50 to 100 word caption, and the load-bearing detail is that the best recipes mix synthetic and original captions instead of replacing one with the other, because pure synthetic text launders away proper nouns and world knowledge. Stage three is tiny by comparison, LLaVA-1.5’s 665K instruction mixture growing to roughly 760K in LLaVA-1.6 with DocVQA, ChartQA and AI2D added, and it buys instruction following, short grounded answers, OCR and chart reading rather than new visual knowledge. The three stages differ by four orders of magnitude in scale and by roughly the same factor in cost per example, which is why the filtering stage is optimised for throughput and the instruction stage for mixture ratios.

(1) Cascade Order Is Cost Order: run cheap deterministic filters (decode check, resolution, aspect ratio, caption length, exact and near-duplicate hashing, NSFW and PII removal) before any model forward pass, since a CLIP score on 12.8B pairs is the single most expensive step in the pipeline.
(2) CLIP Score Is A Precision Knob: the cosine gate raises image-text agreement but systematically deletes long compositional captions, rare entities and text-heavy images, so a threshold tuned for zero-shot classification quietly damages OCR and document tasks.
(3) Cluster Balancing Beats Raw Score: DataComp’s winning filtering-track recipe intersects the CLIP-score gate with an image-embedding cluster filter, keeping pairs whose visual cluster resembles curated concept distributions, which fixes the head-heavy topical skew of the crawl.
(4) Re-Captioning Changes The Supervision, Not The Images: the same 1.4B images are re-labelled by a captioner VLM, so the corpus gains dense spatial, attribute and relational description at roughly the cost of one VLM forward pass per image and zero new crawling.
(5) Mix, Do Not Replace: a mixing probability of about $\alpha \approx 0.8$ synthetic to 0.2 original, or an LLM fusion of both strings as in CapsFusion, retains the named entities and factual hooks that only alt-text carries.
(6) Instruction Data Is Ratio-Sensitive, Not Scale-Sensitive: at the 665K scale the composition (VQA, OCR, region grounding, text-only chat) matters far more than the count, and dropping the text-only share collapses multi-turn conversational quality while adding no visual skill.
(7) Decontamination Is Mandatory: near-duplicate removal against the images and questions of VQAv2, TextVQA, MMMU and friends must run at every stage, because a captioner trained on benchmark-adjacent data will otherwise leak answers into the pre-training corpus.

Three-band pipeline diagram: band A shows a raw Common Crawl pool of 12.8B image-text pairs passing through safety and PII removal, basic resolution and caption-length filters, a CLIP cosine gate keeping the top 30 percent, and cluster balancing plus decontamination to yield 1.4B pairs; band B shows a captioner VLM trained on 100K high-quality captions producing dense synthetic captions, an LLM fusing alt-text with the synthetic caption, and a mixed corpus at alpha near 0.8; band C shows projector alignment on 558K pairs, caption plus interleaved pre-training, visual instruction tuning on 665K to 760K examples, and evaluation on VQAv2, TextVQA, DocVQA and MMMU

Figure 1: The pipeline as three chained corpora rather than one dataset. Band A is a cheap-to-expensive filter cascade that discards about 89% of the crawl, band B re-labels the survivors with a captioner VLM and fuses the result with the original alt-text, and band C spends a four-orders-of-magnitude smaller budget on alignment then instruction tuning.

The filtering stage is best understood as trading recall for precision under a fixed compute budget. DataComp’s central result is that the winning entry is not the largest pool but the most aggressively filtered one: at a fixed number of training samples seen, a 1.4B subset beats the 12.8B pool it came from, because gradient steps spent on mismatched pairs are worse than wasted. The failure mode of that logic is that the CLIP scorer used to filter was itself trained on similarly filtered data, so its notion of “matching” is circular and biased against exactly the long, unusual, or text-dense captions that document and chart understanding require. Production pipelines therefore keep separate sub-pools with different thresholds, plus an explicitly retained OCR-heavy shard, rather than applying one global $\tau$ to everything.

Mathematical Formulation:
$s(I,T) = \cos(f_I, f_T)$
$D_1 = \{(I,T) \in D_0 : s(I,T) \geq \tau\}$
$D_2 = D_1 \cap C_{\mathrm{clust}}$
$T' \sim p_{\phi}(T \mid I)$
$P(\tilde{T} = T') = \alpha$
$\mathcal{L} = -\sum_{t \in A} \log p_{\theta}(y_t \mid I, x, y_{1:t-1})$

Where:

$s(I,T)$ is the CLIP cosine between the image embedding $f_I$ and caption embedding $f_T$ , and $\tau$ is the gate, historically about $0.28$ for a ViT-B/32 scorer or a percentile such as the top 30%.
$D_0$ is the raw pool (12.8B pairs in CommonPool), $D_1$ the CLIP-gated set, and $D_2$ the final corpus after intersecting with the cluster filter $C_{\mathrm{clust}}$ and benchmark decontamination, giving roughly $1.4 \times 10^{9}$ pairs.
$p_{\phi}$ is the captioner VLM and $T'$ the dense synthetic caption it samples for image $I$ ; $\phi$ is trained on a small human or GPT-4V-labelled seed set, typically around 100K captions.
$\alpha \in [0,1]$ is the mixing probability of using the synthetic caption instead of the original alt-text $T$ for a given training sample; $\alpha = 1$ is pure synthetic and $\alpha = 0$ is the raw web baseline.
$\mathcal{L}$ is the instruction-tuning objective over the answer token set $A$ only, with the image $I$ and instruction $x$ as context and loss masked on the prompt, which is what prevents the model from learning to hallucinate its own questions.

The instruction stage also decides the inference bill, because resolution enters through the token count rather than the parameter count. LLaVA-1.5 uses a CLIP ViT-L/14 at 336 px, so each image becomes $(336/14)^2 = 576$ visual tokens, while LLaVA-1.6’s AnyRes scheme tiles a high-resolution image into four crops plus a global thumbnail, giving $5 \times 576 = 2880$ tokens. That five-fold increase is what unlocks DocVQA and ChartQA, and it also means the instruction mixture must contain enough high-resolution document data to justify the tokens, otherwise the model pays the cost without learning to use the detail.

Line chart with synthetic caption mixing ratio alpha on the x axis from 0 to 1 and relative benchmark score on the y axis: a retrieval and captioning curve rises monotonically from 100 to about 118, an entity and world-knowledge curve rises to a peak near alpha 0.65 then falls back to about 100 at alpha 1, and their average peaks near alpha 0.8, marked by a vertical dashed line labelled common operating point

Figure 2: Why re-captioning is a mixture and not a replacement. Synthetic captions monotonically improve retrieval and description because they are fluent and pixel-grounded, but entity and knowledge accuracy peaks well before $\alpha = 1$ and then decays, since a captioner cannot invent the proper nouns, brands, and dates that only human alt-text supplied. The reported operating point in VeCLIP, CapsFusion and Recap-DataComp ablations lands near $\alpha \approx 0.8$ .

Property	Filtered web pairs	Synthetic re-captions	Visual instruction data
Typical scale	1B to 5B pairs after filtering 10B+ raw	100K seed captions, then 1M to 1.3B generated	665K in LLaVA-1.5, about 760K in LLaVA-1.6
Source	Common Crawl alt-text, HTML attributes	A captioner VLM run over already-filtered images	Academic VQA/OCR/grounding sets plus LLM-written dialogue
Cost per example	Crawl plus one CLIP forward pass	One VLM generation of 50 to 100 tokens	Human annotation or strong-model distillation, orders of magnitude higher
What it teaches	Concept coverage, entities, long-tail visual vocabulary	Dense attributes, spatial relations, fluent grounded description	Answer format, instruction following, refusal and multi-turn behaviour
Dominant failure mode	Mismatched or page-level captions, topical head skew	Hallucinated details, lost proper nouns, uniform caption style	Wrong mixture ratio, benchmark overfitting, short-answer bias
Where it enters training	Projector alignment and large-scale pre-training	Mixed into the same pre-training stream at ratio $\alpha$	Final supervised stage, full LLM and projector unfrozen

Transformer VLM

DL0143 VLM Pretraining Data Curation

Comments

Leave a Reply Cancel reply

More posts

MSD0064 Notification Ranking System

MSD0063 Image Copyright Detection

MSD0062 Ads Ranking System

MSD0061 Nearby Place Recommendation