DL0128 Subword Regularization

Subword regularization is the practice of training a translation model on multiple stochastically sampled segmentations of the same sentence instead of the one deterministic segmentation a tokenizer normally emits. Any subword vocabulary is ambiguous: the word “smaller” can be a single piece, or small plus er, or sm plus all plus er, and a standard tokenizer always returns the same choice, so the model never sees the alternatives. Two mechanisms implement the sampling: unigram-LM sampling (Kudo 2018, shipped in SentencePiece), which draws a path from the segmentation lattice with probability proportional to , and BPE-dropout (Provilkov 2020), which keeps the ordinary BPE merge table but skips each merge with probability during application. The quality gain has three sources: it is free data augmentation over the same parallel corpus, it trains the embeddings of rare pieces that a deterministic segmenter almost never produces, and it makes the encoder robust to the segmentation shifts caused by typos, morphology, and domain change. Reported gains are roughly 1 BLEU on average, shrinking toward noise on large in-domain high-resource benchmarks and reaching 2 BLEU or more on low-resource pairs and out-of-domain test sets. Sampling is a training-time-only device: decoding uses the deterministic segmentation, or optionally averages scores over sampled segmentations of the source.

(1) Segmentation Is Ambiguous: a fixed vocabulary admits exponentially many segmentations per sentence, and deterministic tokenization silently commits to one of them in every epoch.
(2) Unigram-LM Sampling: SentencePiece samples from the -best lattice paths with a temperature , typically and near 0.1 to 0.2, where small is near-uniform and recovers the Viterbi segmentation.
(3) BPE-Dropout: drop each merge operation independently with on both source and target; the vocabulary and merge table are unchanged, so it is a drop-in change to the tokenizer call.
(4) Approximate Marginalization: the model is pushed to score a sentence pair well under any plausible segmentation, which is closer to the true marginal than a single-path proxy.
(5) Robustness Beats In-Domain Accuracy: the largest wins appear on low-resource, morphologically rich, noisy, and out-of-domain data, exactly where the test-time segmentation drifts from the training-time one.
(6) Costs Are Real: average sequence length grows, tokenization moves into the data loader and cannot be cached, and convergence is slower in wall-clock terms for the same number of updates.

Figure 1: One word, three legal paths through the segmentation lattice. Deterministic tokenization always returns the top path, so the -er morpheme boundary is never shown to the encoder; sampling at returns the top path only about half the time and spends the rest of the probability mass on the finer decompositions.

The reason the same trick helps two different segmenters is that both are inverting a latent variable the corpus never labels. A unigram language model gives an explicit probability to each path, so its lattice can be sampled cleanly and the temperature gives a continuous dial from “uniform over the lattice” to “deterministic”. BPE has no probability model at all, only an ordered merge list, so BPE-dropout injects randomness procedurally instead: skipping a merge with probability produces a shorter piece and therefore a longer, more character-like segmentation, and stacking many independent skips over a long word yields a wide distribution of segmentations. Both settings need a knob that is gentle: too much randomness pushes segmentation toward characters, inflating sequence length, wasting attention on re-learning spelling, and destroying the frequency statistics that made the vocabulary useful in the first place.

Mathematical Formulation:

Where:

• and are the raw source and target sentences, while and are particular segmentations of them into vocabulary pieces.
• is the unigram probability of piece , estimated by EM when the SentencePiece unigram vocabulary is built, and is the number of pieces on that path.
• is the smoothing exponent of the sampling distribution: gives a near-uniform draw over the -best lattice paths and collapses to the single Viterbi path, so deterministic tokenization is the limiting case.
• The expectation in is taken over and , approximated by one fresh sample per example per epoch rather than by an inner loop.
• is the true segmentation-marginalized translation probability that the training objective approximates; the sum over is intractable, which is exactly why sampling is used.
• is the number of source segmentations used in optional n-best decoding, where with the Viterbi path is the standard, cheapest inference setting.

Figure 2: The temperature controls exactly two things that matter. Left: the entropy of the sampling distribution, which is how much augmentation diversity the model receives per epoch. Right: the expected pieces per word, which is the sequence-length tax you pay for that diversity. Beyond roughly both collapse to the deterministic Viterbi behaviour and the method does nothing.