Free-Prompt-Editing: Efficient Text-Guided Image Editing with Stable Diffusion

Alibaba Cloud AI Platform PAI, together with Prof. Jia Kui's team at South China University of Technology, presented the Free‑Prompt‑Editing (FPE) algorithm at CVPR 2024, revealing how Stable Diffusion can be exploited for high‑efficiency image editing.

Background

Text‑to‑Image synthesis models such as Stable Diffusion, DALL‑E 2, and Imagen have achieved remarkable success in generating realistic images from textual prompts and have attracted significant academic and industrial interest. Beyond generation, these models possess strong text‑guided image editing capabilities, making it crucial to understand and harness their attention mechanisms for reliable editing.

Attention Map Probe Analysis

The authors pose the question whether attention maps in diffusion models contain only weighting information or also embed semantic image features. Using probe classifiers, they show that both cross‑attention and self‑attention maps encode meaningful category information.

Probe Results and Conclusions

Editing cross‑attention maps is optional; replacing them can cause failure.

Cross‑attention maps carry semantic features of the conditioning tokens, so swapping source and target maps may produce unexpected results.

Self‑attention maps are critical for text‑guided image editing because they preserve spatial relationships and structural details.

Experiments demonstrate that selective replacement of attention maps at different layers leads to varying editing outcomes, confirming the importance of self‑attention in preserving image structure.

Algorithm

Based on the probe findings, FPE directly replaces self‑attention maps of the source image with those of the target prompt in diffusion layers 4–14. For synthetic image editing, the source layout is merged with target semantics; for real‑image editing, DDIM‑inversion reconstructs latent features before applying the self‑attention replacement. The method benefits from cross‑attention for prompt‑image alignment without needing source cross‑attention maps.

Figure 4 illustrates the FPE framework for synthetic image editing.

Experimental Results

FPE successfully edits attributes, styles, scenes, and categories of both synthetic and real images, outperforming state‑of‑the‑art methods on Wild‑TI2I and ImageNet‑R‑TI2I benchmarks, especially in the CDS metric, while maintaining a good trade‑off between speed and effectiveness.

Figure 6 shows diverse editing results, including gender conversion, age transformation, hairstyle changes, background replacement, and category swaps.

References

Rombach R, et al. High‑resolution image synthesis with latent diffusion models. CVPR 2022.

Hertz A, et al. Prompt‑to‑Prompt image editing with cross‑attention control. arXiv 2022.

Brooks T, et al. InstructPix2Pix: Learning to follow image editing instructions. CVPR 2023.

Cao M, et al. Masactrl: Tuning‑free mutual self‑attention control for consistent image synthesis and editing. ICCV 2023.

Tumanyan N, et al. Plug‑and‑play diffusion features for text‑driven image‑to‑image translation. CVPR 2023.

Park D H, et al. Shape‑guided diffusion with inside‑outside attention. WACV 2024.