Design a distributed data preprocessing pipeline for training large language models at Amazon. The Bedrock team needs to prepare training corpora from petabytes of raw text (web crawls, code repositories, internal documents, licensed datasets) by tokenizing, deduplicating, filtering for quality, removing harmful content, and sharding into training-ready chunks.
The pipeline must process data arriving in S3 and other stores, handle near-duplicate detection at scale (MinHash, SimHash, exact hash), apply quality classifiers to remove low-quality text, run PII and safety filters, and output sharded tokenized sequences ready for distributed training. It must process hundreds of terabytes per run, be resumable from checkpoints, and produce reproducible datasets with full provenance tracking.
How would you design this system? Cover the ingestion architecture (batch S3 versus streaming), the deduplication strategy (exact versus near-duplicate at scale), the quality and safety filtering pipeline, the tokenization and sharding strategy, and how you ensure reproducibility and provenance across pipeline runs.
The Problem: half a petabyte of messy text arrives per run, most of it duplicated, boilerplate, unsafe, or machine-generated, and the training cluster needs a byte-identical, provenance-tracked shard set within days.
Answer
The design is an immutable, content-addressed data lake on S3 with a cheap-to-expensive filter cascade running as batch Spark or Ray jobs. Every stage reads a frozen snapshot, appends scores instead of deleting rows, and writes a new layer whose path is keyed by the hash of its code plus config, which is what makes runs reproducible and resumable. Ingestion is batch over snapshots rather than streaming, because deduplication is a global operation that needs a full-corpus view. Two decisions are pivotal. First, three-tier deduplication (exact document and line hashes, then MinHash with LSH banding for near duplicates, then substring dedup on high-risk subsets) runs before the model-based quality classifier, so the expensive scorer sees roughly a third of the documents. Second, tokenization is deferred to the last stage so the durable artifact stays text, which means a tokenizer or vocabulary change costs a few hours of CPU rather than a full pipeline rerun.
(1) Immutable Snapshot Ingestion: crawls and repos land in a bronze layer as append-only, content-addressed objects with a snapshot manifest; streaming is used only to buffer arrivals, never to feed the transforms.
(2) Three-Tier Deduplication: exact SHA-256 on documents and lines kills boilerplate, MinHash plus LSH bands catch near duplicates, and union-find keeps one canonical document per cluster.
(3) Score, Never Delete: heuristics, quality classifiers, and safety and PII detectors write scores into columnar tables, and a single declarative policy file turns scores into keep/drop decisions per run.
(4) Cost-Ordered Cascade: language ID and cheap heuristics first, then dedup, then the learned quality classifier, then safety and PII, so each expensive model runs on the smallest surviving corpus.
(5) Tokenize Last, Pack Fixed: BPE tokenization emits packed 8192-token sequences with document-boundary masks into roughly 40,000 one-gigabyte shards plus an index for deterministic, resumable loading.
(6) Provenance And Resumability: a per-document lineage table plus a partition-level task ledger give a Merkle dataset version, replay of only failed partitions, and targeted rebuilds for takedown requests.
Figure 1: The cascade: cheap normalization and dedup shrink the corpus before any learned model runs, tokenization is the final derived stage, and every stage writes lineage into the provenance store that defines the dataset version.
Clarify Before Designing:
(1) Token Target: how many training tokens does the model plan need, and for how many epochs? A 10T-token target with single-epoch training lets us filter aggressively; a 2T-token budget with repetition does not.
(2) Licensing And Compliance: which sources are licensed for training, what is the right-to-delete SLA, and are there regional data-residency constraints on where the pipeline may run?
(3) Cadence: is this a one-off corpus build or a monthly refresh on new crawl snapshots? Incremental refresh forces us to persist MinHash signatures for cross-run dedup.
(4) Tokenizer Stability: is the vocabulary frozen, or still being trained? That decides whether tokenized shards or filtered text is the durable artifact.
(5) Mixture Requirements: what are the target per-domain and per-language proportions (code, multilingual, internal docs), since mixture weights drive per-slice keep-rate floors.
(6) Budget: what wall-clock and dollar budget per run, and how many proxy-model ablations can we afford to validate a corpus change?
Leave a Reply