DL0105 DiT: Diffusion Transformers

The core innovation is mostly subtractive: DiT deletes the multi-scale convolutional U-Net and replaces it with a plain ViT-style transformer that runs on a flat sequence of latent patch tokens at constant resolution and constant width, with no downsampling, no upsampling, and no encoder-decoder skip connections. A latent from an 8x VAE is cut into non-overlapping patches ( gives 256 tokens), linearly projected to width , given positional embeddings, and pushed through identical blocks. The only diffusion-specific machinery left is how conditioning enters: adaLN-Zero regresses a per-block scale, shift, and residual gate from the summed timestep and class embeddings, with the gate initialized to zero so each block starts as an identity map. Because the backbone is isotropic, sample quality becomes a smooth function of transformer Gflops rather than of hand-tuned channel schedules, and DiT-XL/2 (28 blocks, width 1152, 675M parameters, 118.6 Gflops) reached FID 2.27 with classifier-free guidance on ImageNet after 7M training steps, ahead of the U-Net LDM and ADM baselines. That predictable scaling, plus the ability to reuse ordinary transformer infrastructure, is why later systems such as Stable Diffusion 3 and video generators adopted DiT-style backbones.

(1) Isotropic Token Stack: patchify once, then keep the sequence length and hidden width fixed through every block, so there is no multi-scale hierarchy and no skip connections to carry high-frequency detail.
(2) adaLN-Zero Conditioning: timestep and class are injected by modulating LayerNorm and gating the residual branch instead of through cross-attention or extra input channels, and the zero-initialized gate makes a 28-block stack start as the identity.
(3) Patch Size Is A Compute Knob: halving quadruples the token count and roughly quadruples backbone Gflops without adding parameters, which makes compute and capacity independently tunable.
(4) Quality Tracks Gflops: FID decreases monotonically with backbone Gflops across model sizes and patch sizes, so a smaller model with small patches can beat a larger model with large patches.
(5) Infrastructure Reuse: the backbone is a standard transformer, so FlashAttention, sequence and tensor parallelism, and spatiotemporal patching for video all transfer directly from the LLM and ViT ecosystem.

Figure 1: The U-Net spends parameters on a resolution pyramid with skip connections, while DiT keeps one token sequence at fixed width and pushes all conditioning through adaLN-Zero modulation of each block.

The conditioning choice was not incidental. The DiT paper ablated four options at matched compute: in-context conditioning (append timestep and class as extra tokens), cross-attention to a two-token condition sequence, plain adaptive LayerNorm, and adaLN-Zero. The ranking was consistent, with adaLN-Zero best, then adaLN, then cross-attention, then in-context, and adaLN-Zero also added the fewest Gflops because it needs no extra tokens or extra attention operation. The zero-initialized gate matters because a deep residual stack whose blocks all start as identity behaves like a shallow network early in training, which is the same trick that stabilizes very deep ResNets and ViTs. The cost of dropping skip connections is that all high-frequency reconstruction has to be learned inside the token stack and by the VAE decoder, which is one reason DiT operates in a compressed latent space rather than on raw pixels.

Mathematical Formulation:

Where:

• is the token count after patchifying a latent of spatial size with patch size ; for and this gives .
• is the pooled conditioning vector built from the diffusion timestep and the class or text embedding .
• are the scale and shift applied to the normalized activations of sub-layer , and is the residual gate; indexes the attention and feed-forward sub-layers.
• is the token sequence, is LayerNorm without learnable affine parameters, and is multi-head self-attention (the feed-forward sub-layer uses identically).
• is the per-layer attention cost, quadratic in and therefore quartic in the inverse patch size.
• Required initial condition: at step 0, so every block is an identity map and the residual stream passes through unchanged.

Figure 2: Approximate FID against backbone Gflops at a fixed 400K-step training budget and without classifier-free guidance, which is why the absolute values sit far above the guided FID 2.27 quoted for the fully trained DiT-XL/2. What matters here is the trend: quality tracks compute rather than parameter count, so DiT-B with can outrun DiT-L with despite having far fewer parameters.

Property	DiT	U-Net (ADM, LDM, SDXL)
Spatial handling	One fixed-length token sequence, no resampling and no skips	Resolution pyramid 32 → 16 → 8 → 4 → 8 → 16 → 32 with skip connections
Conditioning path	adaLN-Zero: per-block scale, shift, and zero-initialized residual gate from	Timestep embedding added inside ResBlocks, text or class via cross-attention
Compute knobs	Depth, width, and patch size; patch size changes Gflops at constant parameters	Channel multipliers, blocks per level, and which levels get attention
Inductive bias	Minimal beyond positional embeddings, so it needs more data and compute to pay off	Strong locality and multi-scale bias, sample-efficient at small budgets
Cost as resolution grows	attention, so latent compression and patch size are load-bearing	Convolutions grow linearly in pixels; attention only at low-resolution levels
Scaling behavior	FID a smooth decreasing function of Gflops, easy to extrapolate	Gains depend on hand-designed schedules and saturate less predictably

One caveat when quoting numbers in an interview: FID is only comparable within a fixed evaluation protocol. The headline 2.27 comes from the fully trained DiT-XL/2 sampled with classifier-free guidance at scale 1.5; the same checkpoint sampled without guidance lands near 9.6, and the scaling sweep above uses a much shorter 400K-step budget with no guidance at all. Guidance, training steps, sampler, and step count all move the number by more than the architectural gap being discussed, so state the protocol alongside the score.