DL0106 Stable Diffusion Architecture

Stable Diffusion is three separately trained networks plus one training-free sampler, and only one of the three is trained on the diffusion objective. A frozen KL-regularized VAE maps a image to a latent and back; a frozen CLIP text encoder turns the prompt into a sequence of token embeddings; and a conditional U-Net (860M parameters in SD 1.5) predicts the noise present in a noisy latent, given the timestep and the text context. At generation time the sampler starts from on the latent grid and repeats a fixed loop: call the U-Net, combine the conditional and unconditional predictions with classifier-free guidance, then let the scheduler (DDIM, Euler, DPM-Solver) take one step of the reverse process. After 20 to 50 such steps the VAE decoder is called exactly once to turn the final latent into pixels, so the full path is encode → denoise → decode with the text encoder feeding every denoising step through cross-attention.

(1) VAE Codec: a convolutional encoder and decoder pair, frozen after stage-one training with an L1 plus LPIPS plus patch-GAN objective, that removes imperceptible high-frequency detail so the denoiser works on 16,384 values instead of 786,432.
(2) Text Encoder: CLIP ViT-L/14’s transformer, frozen, producing per-token hidden states rather than a single pooled vector, which is what makes word-level prompt control possible.
(3) U-Net Denoiser: the only trained diffusion component, built from ResBlocks that receive a sinusoidal timestep embedding through a learned projection added to their feature maps, interleaved with transformer blocks.
(4) Cross-Attention Is The Conditioning Interface: queries come from the latent feature map, keys and values come from the text context, so the prompt influences the image at 16 separate points in SD 1.5’s U-Net.
(5) Scheduler: not a network at all but a numerical solver for the reverse SDE or probability-flow ODE, which is why you can swap samplers and step counts on a trained checkpoint without retraining.
(6) Classifier-Free Guidance: each step runs the U-Net twice, once with the prompt and once with the empty string, and extrapolates between them, which doubles the per-step cost and is the main knob for prompt adherence.

Figure 1: The loop is the architecture. The U-Net is evaluated twice per step for guidance, the text encoder runs once for the whole generation, and the decoder runs once at the end; everything inside the loop stays on the small latent grid.

Inside the U-Net the latent descends a resolution ladder, , with channel widths 320, 640, 1280, 1280, and climbs back up with skip concatenations from the matching encoder level. Attention is deliberately not applied everywhere: in SD 1.5 the transformer blocks sit at the , , and levels plus the mid block, and the deepest level is pure convolution, because self-attention cost grows as in the token count and the highest-resolution level already carries 4,096 tokens. SDXL rebalances exactly this: it drops attention at the finest level entirely and stacks far more transformer blocks at and , which is how it reaches 2.6B parameters while remaining trainable at . Each transformer block is self-attention, then cross-attention, then a GEGLU feed-forward, so spatial coherence and prompt alignment are handled by two different mechanisms in the same block.

Figure 2: The SD 1.5 U-Net applies self-attention and cross-attention at three of four resolutions and leaves the deepest level convolution-only; skip concatenations carry high-frequency structure past the bottleneck so the decoder path can restore fine detail.

Mathematical Formulation:

Where:

• is the latent at timestep , and is the decoded image.
• is the flattened spatial feature map entering a transformer block, and is the frozen CLIP context of shape .
• are the per-layer projections and is the head dimension; queries carry image content while keys and values carry text.
• and are the conditional and unconditional noise predictions, and is the guidance scale (typically ).
• is the scheduler update (DDIM, Euler, DPM-Solver), applied for descending over the chosen 20 to 50 timesteps.
• is the frozen VAE decoder and the latent scaling constant; the required initial condition is , which only holds because normalizes the latent variance.