DL0103 Stable Diffusion Latent Space

Latent diffusion splits image generation into two stages that are trained separately: a convolutional autoencoder first compresses pixels into a small spatial latent, and the diffusion model then learns the noise process entirely inside that latent grid. Stable Diffusion uses a KL-regularized VAE with downsampling factor , so a image becomes a latent: 786,432 values collapse to 16,384, a 48x reduction in the tensor the U-Net has to denoise. The motivation is that the two stages solve different problems. Pixel-space diffusion wastes most of its capacity and most of its training steps modeling imperceptible high-frequency detail, exactly the information a perceptual codec throws away for free; the autoencoder handles that perceptual compression once, and the diffusion model is left with the semantic compression problem of arranging layout, objects, and style. The compute argument is even stronger than the 48x suggests, because the U-Net contains self-attention: token count drops from 262,144 to 4,096, so any attention block gets times cheaper. That is what turned high-resolution text-to-image training from a large-cluster project into something reproducible on modest hardware, and what makes 50-step sampling on a consumer GPU feasible at all.

(1) Two-Stage Factorization: the autoencoder is trained first and then frozen; the diffusion model never sees a pixel, and the decoder is invoked exactly once at the end of sampling.
(2) Quadratic Savings On Attention: the latent grid has fewer positions, so convolutions get cheaper and self-attention gets roughly cheaper.
(3) It Is Not A Generative VAE: the KL weight is tiny (on the order of ) and the reconstruction loss adds LPIPS plus a patch discriminator, so the encoder behaves as a sharp lossy codec rather than a prior you would ever sample from.
(4) Latent Scaling Matters: encoder outputs are multiplied by a constant (0.18215 for SD 1.x, 0.13025 for SDXL) so the latent has roughly unit variance and the standard noise schedule reaches a true terminal state.
(5) Cheap Conditioning And Editing: CLIP text embeddings enter through cross-attention at every latent resolution, and img2img, inpainting, and ControlNet all operate on the same small latent grid.
(6) The Decoder Is The Fidelity Ceiling: nothing the diffusion model does can recover detail the 4-channel latent discarded, which is why small text, fine textures, and tiny faces degrade first.

Figure 1: Training and sampling both live in the 4,096-token latent grid; the frozen encoder and decoder are the only components that ever touch the 262,144-pixel image, and text conditioning enters the denoiser through cross-attention.

The choice of is a genuine trade-off rather than a free win. The original LDM ablations show that is the sweet spot: at or training is slow because the model is still doing perceptual compression itself, while at with only 4 channels the autoencoder becomes the bottleneck and sample quality saturates no matter how long the diffusion model trains. Channel count is the other lever, and it is the one that later models moved: SD3 and Flux keep but raise the latent to 16 channels, quartering the compression ratio from 48 to 12 to trade a little step cost for a much better reconstruction ceiling on text and fine structure. A practical consequence of the frozen-codec design is that latents are model-specific: an SDXL latent decoded by an SD 1.5 decoder produces color-shifted garbage, because the two autoencoders were trained with different scaling and channel statistics.

Mathematical Formulation:

Where:

• is the image and its reconstruction; is the clean latent with and .
• and are the frozen VAE encoder and decoder, and is the latent scaling constant that normalizes the variance.
• is the timestep, the cumulative noise schedule, and the sampled Gaussian noise.
• is the latent denoiser (U-Net in SD 1.x and SDXL, a transformer in DiT-style successors), and is the text-encoder output injected by cross-attention.
• is the spatial downsampling factor and the latent channel count; is the element compression ratio, which is 48 for and 12 for .
• Required initial condition at sampling: on the latent grid, which only holds if was applied and .

Figure 2: Because self-attention scales as in the token count, compressing by cuts attention cost by ; the setting used by Stable Diffusion buys a 4096x reduction while keeping the decoder’s reconstruction error acceptable.