How VICTORIA Boosts Text‑Guided Image Editing with Language‑Aware Diffusion

Alibaba Cloud AI Platform PAI team announced their image editing algorithm paper at ACM MM2024, marking significant academic recognition for their work. Text‑to‑image synthesis (TIS) has become a key frontier at the intersection of computer vision and natural language processing, enabling images to be generated from textual descriptions. Training‑free text‑guided image editing (TIE) uses pretrained TIS models to edit images via simple textual prompts, allowing operations such as color changes, object addition or removal, and style transfer without specialized software.

Existing TIE methods struggle with multi‑object editing, often losing objects, attributes, or background details, as illustrated in Figure 1.

The proposed VICTORIA algorithm addresses these issues by incorporating linguistic knowledge into the editing process. It parses dependency relations between words in the input edit text and reflects them in the intermediate representations of the attention layers, thereby correcting and generating the target image.

VICTORIA Framework

Figure 2 shows the overall architecture. First, VICTORIA controls the self‑attention mechanism to maintain spatial consistency between the original and edited images. Next, it analyzes word dependencies in the edit prompt and actively intervenes in the cross‑attention maps during generation, improving the fidelity of edited regions. Finally, it uses the cross‑attention map to mask unchanged parts of the image, preserving original content.

VICTORIA Pseudocode

Figure 4 demonstrates VICTORIA's editing results, successfully modifying multiple objects' attributes, styles, scenes, and categories within the same image.

Figure 5 compares VICTORIA with other state‑of‑the‑art image editing techniques on both real photos and synthetic images. VICTORIA consistently achieves fine‑grained edits that closely match textual descriptions while preserving structural details of the original image.

The source code of VICTORIA has been contributed to the EasyNLP framework, inviting researchers and practitioners to explore and build upon the algorithm.