DL0177 Discrete vs Continuous Visual Tokens

Both paths begin identically, with a convolutional or ViT encoder turning the image into a grid of -dimensional vectors. The only structural difference is whether a quantizer follows. A discrete tokenizer (VQ-VAE, VQ-GAN) snaps each vector to its nearest entry in a learned codebook of vectors and keeps only the integer index, so an image becomes a string of ids over a finite vocabulary that a jointly trained decoder can invert back to pixels. A continuous representation (ViT patch outputs, CLIP or SigLIP features, KL-VAE latents) keeps the float vector, which preserves far more information but has no finite support, so no softmax and no cross-entropy can be defined over it. That single difference decides the downstream interface: discrete ids plug into next-token or masked-token prediction with exactly the machinery used for text, while continuous features must be projected into an LLM or denoised by a diffusion model. As a default, understanding prefers continuous features because quantization discards the high-frequency detail that OCR and fine-grained VQA depend on, while generation historically preferred discrete tokens because a categorical likelihood is easy to train and easy to sample.

(1) Only The Quantizer Differs: the encoder, the patch grid, and the spatial downsampling factor can be identical; adding a nearest-code lookup converts a float grid into an id grid.
(2) Information Budget: a discrete token carries bits (10 to 18 in practice), while a continuous patch vector carries roughly bits of activation, three orders of magnitude more.
(3) Gradient Path: is non-differentiable, so VQ needs a straight-through estimator plus codebook and commitment losses, whereas continuous encoders train by plain backpropagation.
(4) Reconstruction Cost: heavy compression makes plain L2 reconstruction blurry, which is why VQ-GAN adds perceptual and patch-GAN losses to keep 16x-downsampled decodes sharp.
(5) Downstream Interface: discrete gives one softmax vocabulary shared with text; continuous gives features for a projector, cross-attention, or a latent diffusion denoiser.
(6) Task Split: continuous features dominate VLM understanding benchmarks; discrete tokens dominate when the goal is a single unified next-token model that also emits pixels.

Figure 1: The same encoder, two endings. Deleting the quantizer leaves continuous features that a projector feeds to an LLM; inserting it buys a finite vocabulary and a pixel decoder at the price of bits per token. Note that the understanding path has no decoder at all, which is why an understanding-only encoder is never required to be invertible.

The practical difficulty of discrete tokenization is that the codebook must be learned through a non-differentiable lookup. The straight-through estimator simply copies the decoder gradient past the quantizer, which is a biased estimate that works only if the encoder output stays close to its assigned code, hence the commitment loss. The characteristic failure is codebook collapse: a few entries win most assignments, the rest receive no gradient and die, and effective vocabulary size stops tracking nominal . Standard mitigations are EMA codebook updates, low-dimensional -normalized codes, dead-code re-initialization, and an entropy bonus on the assignment distribution. Continuous encoders have none of this machinery, but they also cannot be sampled from, since there is no distribution over that a softmax can express, which is precisely why continuous-latent generation requires a diffusion or flow model rather than a token classifier.

Mathematical Formulation:

Where:

• is the input image, the encoder, and one continuous patch vector from the encoder grid; keeping and stopping here is the continuous path.
• for are the learned codebook vectors, is the selected index (the actual token), and is the vocabulary size.
• is the quantized vector fed to the decoder , and the residual is information the model can never recover.
• is the stop-gradient operator and (typically ) weights the commitment loss that pulls encoder outputs toward their assigned codes.
• The full VQ-GAN objective adds a reconstruction term, an LPIPS perceptual term, and a patch-discriminator term to ; only the reconstruction term survives in a plain VQ-VAE.

Bit Budget For One 256×256 Image:

All four numbers are in bits. A 16x-downsampling tokenizer with compresses the image to 256 ids of 14 bits each, about 448 bytes, while a ViT-L/14 tower keeps 256 patch vectors of 1024 bf16 activations, about 512 KiB. That ratio is the whole argument: it is why a discrete sequence is short enough to model autoregressively alongside text, and equally why an OCR-heavy or chart-reading task should not be routed through it.

Figure 2: Left, quantization is a Voronoi partition of the latent space: the id names the cell, and the offset inside the cell is thrown away. Right, approximate published reconstruction results show that a plain VQ codebook stops improving past about entries because of codebook collapse, whereas lookup-free and FSQ-style quantizers keep scaling toward the continuous-latent floor.