Self‑Swap Guidance: Boost Diffusion Models by Direct Token Swaps (CVPR 2026 Oral)

Improving the generation quality of diffusion models traditionally relies on guidance during inference, which either depends on text prompts (e.g., Classifier‑Free Guidance) or adds explicit noise, both of which have notable drawbacks such as the need for specialized training or limited effectiveness.

The authors propose Self‑Swap Guidance (SSG), a novel paradigm that requires no text, adds no noise, and leaves the pretrained model unchanged. The key idea is to introduce a structural perturbation by swapping the most dissimilar token features in both spatial and channel dimensions during the forward pass, thereby creating a “degraded” prediction path that can be used for reverse guidance.

Implementation uses two parallel inference branches. One branch outputs the original noise prediction ϵ_ori unchanged. The second branch performs a spatial self‑swap followed by a channel self‑swap on selected token or channel pairs—specifically the pairs with the lowest similarity—and produces a perturbed noise prediction ϵ_pert. At each diffusion timestep the difference between the two predictions defines a guidance direction, scaled by a coefficient ω, analogous to CFG.

Experiments on unconditional and text‑conditioned generation tasks (COCO‑2014, COCO‑2017, ImageNet) show that SSG consistently surpasses existing unconditional guidance methods such as SAG, SEG, and PAG on metrics including FID, CLIP Score, Inception Score, AES, PickScore, ImageReward, Precision, and Recall. Qualitative results also demonstrate more stable and natural images with better text alignment when conditioning is used.

Ablation studies compare random token swapping with the proposed most‑dissimilar swapping, and spatial versus channel swapping. Random swapping already outperforms prior methods, but the most‑dissimilar strategy yields the highest human‑preference scores. Channel swapping alone performs better than spatial swapping, and combining both achieves the best overall trade‑off between image quality and aesthetic perception.

Further discussion contrasts SSG with contemporaneous work TPG, noting that while both perturb features spatially, SSG’s selective token swapping and additional channel perturbation provide superior guidance quality with comparable computational cost. The authors acknowledge limitations: the method lacks a systematic theoretical foundation, is sensitive to the choice of layers where swaps are applied, and incurs extra computation for similarity calculations. Future work includes developing more robust perturbation mechanisms and optimizing efficiency.