Attention Distillation in Diffusion Models: CVPR 2025 Technique Outperforms Traditional Image Generation

Highlights

Analyzes the limitations of existing plug‑and‑play attention feature methods and proposes a novel attention‑distillation loss that reproduces visual features of a reference image, achieving significantly superior results.

Develops attention‑distillation guided sampling, an improved classifier‑guidance technique that integrates the loss into the denoising process, greatly accelerating synthesis and supporting a wide range of visual‑feature‑transfer applications.

Problem Statement

Current diffusion‑based generators (e.g., Stable Diffusion) excel at modeling complex data distributions but struggle to transfer the visual style or texture of a reference image when using plug‑and‑play attention injection, because the residual connections dilute the injected information.

Proposed Solution

Attention Distillation (AD) Loss

The authors treat the attention output of the reference branch as the ideal stylized representation. They compute an L1 loss between this ideal attention and the attention produced by the target branch:

Optimizing this loss by back‑propagation through the latent space forces the diffusion model to reproduce high‑quality style and texture details that were lost with plain KV injection.

Content Preservation Loss

To keep the semantic content of the target image, a second L1 loss is defined between the query vectors of the target and reference branches:

The combined loss (AD + content) guides the latent code toward both visual style fidelity and content consistency.

Attention‑Distillation Guided Sampling

During DDIM sampling, the AD loss is incorporated as an energy function that modifies the denoising direction at each timestep. The update rule replaces the standard classifier‑guidance term with the gradient of the AD loss, and an Adam optimizer automatically adjusts the guidance strength:

By optimizing directly in latent space, the method avoids the costly image‑space loss computation used by recent works.

Improved VAE Decoding

Because the VAE decoder in latent diffusion models is perceptually lossy, the authors fine‑tune it on the reference image using an L1 reconstruction loss:

This yields sharper textures for high‑frequency tasks such as texture synthesis.

Experiments

Applications and Comparisons

The AD loss is applied to several visual‑feature‑transfer tasks and compared against state‑of‑the‑art baselines.

Style and Appearance Transfer: Compared with CSGO, StyleShot, StyleID, StyTR2, NST, Cross‑Image Attention, and SpliceViT. The proposed method captures high‑quality, consistent styles while preserving semantic structure, and avoids color oversaturation seen in Cross‑Image Attention.

Text‑to‑Image Generation in a Specific Style: Benchmarked against Visual Style Prompting, InstantStyle, and RB‑Modulation. The method matches textual semantics and yields far better style fidelity to the reference image.

Controlled Texture Synthesis: Introduces a mask‑guided variant of AD loss to restrict attention to user‑specified regions, outperforming patch‑based GCD and Self‑Rectification in edge smoothness and artifact reduction.

Texture Expansion: Applies AD‑guided sampling to MultiDiffusion, enabling high‑resolution texture generation (e.g., 512×1536) where prior methods struggle.

Ablation Studies

Two ablations are presented:

Content‑Loss Weight: Varying the weight changes the degree of abstraction in style transfer and the smoothness of identity transition in appearance transfer (see Fig. 11).

Guidance Optimizer: Manual guidance strength often leads to loss of texture or appearance details, whereas an Adam optimizer automatically balances the gradient updates, producing results that closely match the visual features of the reference (see Fig. 12).

User Preference Study

A user study with 50 participants (1,500 responses) evaluated three transfer tasks across six competing methods. The proposed approach received the highest preference scores in all tasks, confirming its qualitative superiority.

Conclusion

The authors present a unified framework for visual‑feature transfer that combines a novel attention‑distillation loss with guided latent‑space sampling and optional VAE fine‑tuning. Experiments across style transfer, text‑to‑image generation, and texture synthesis demonstrate that the method overcomes the limitations of prior plug‑and‑play attention techniques and delivers faster, higher‑quality results.

References

[1] Attention Distillation: A Unified Approach to Visual Characteristics Transfer.