Latent Forcing: Reordering Diffusion Steps Boosts Pixel‑Level Image Quality

Problem

Pixel‑level diffusion models generate high‑fidelity images but often produce structural distortion because high‑frequency texture details interfere with low‑frequency semantic layout during denoising. Latent‑space diffusion models are fast and capture global structure but require a decoder, which introduces reconstruction error and prevents end‑to‑end modeling of raw pixels.

Traditional bottlenecks

In pixel‑wise diffusion the model predicts fine‑grained pixel colors before the overall object outline is clear, violating the natural visual generation order. Latent‑space models compress images into low‑dimensional tokens, enabling rapid generation, but the decoder adds error and loses direct pixel fidelity.

Latent Forcing insight

Reorder the diffusion trajectory so that a latent sketch is generated first, establishing a semantic skeleton, followed by pixel‑level refinement. This mirrors the human process of drafting a sketch before coloring.

Dual‑time variable mechanism

Without changing the Transformer architecture, introduce two independent denoising schedules:

Latent variable first: during early steps the latent tokens are denoised to form a coarse semantic backbone.

Pixel refinement later: after the structure is fixed, pixel tokens are denoised to add fine details.

The token count remains unchanged, so computational overhead is negligible and the process stays end‑to‑end.

Training and inference details

Both latent and pixel tokens are processed simultaneously during training and generation, each following its own noise schedule. The latent sketch is discarded after generation; the final output is a 100 % pixel‑accurate image without any decoder.

Experimental results

On ImageNet‑256 with the same compute budget (80 training epochs), Latent Forcing reduces conditional FID from 18.60 (JiT+REPA) to 9.76. With a ViT‑L model trained for 200 epochs, it achieves guided FID 2.48 and unguided FID 7.2, establishing a new state‑of‑the‑art for pixel‑space diffusion Transformers.

These results contradict the prior belief that higher‑rate lossy compression is required for good FID; Latent Forcing retains 100 % original pixel precision while surpassing lossy baselines.

Reference

Paper: https://arxiv.org/abs/2602.11401