A Survey of Multimodal Image Synthesis and Editing with Generative AI

Generative AI has become a frontier technology in artificial intelligence, widely applied to various visual synthesis tasks. With the release of DALL-E2 , Stable Diffusion and DreamFusion , AI painting and 3D generation have achieved astonishing visual effects and explosive global growth. This survey paper provides answers to how these generative AI methods create realistic visuals and how deep learning and neural networks enable painting, 3D generation, and other creative tasks.

In the first chapter, we describe the significance and overall development of multimodal image synthesis and editing, outline the contributions of this paper, and present its overall structure.

The second chapter introduces common guidance modalities based on the data type: visual guidance, textual guidance, audio guidance, and the recent DragGAN control‑point guidance, along with corresponding data processing methods.

The third chapter classifies existing methods according to model frameworks, including GAN‑based approaches, diffusion models, autoregressive models, and Neural Radiance Fields (NeRF) methods.

Because GAN‑based methods typically use conditional GANs and GAN inversion, the paper further details the fusion of control conditions, model architectures, loss designs, multimodal alignment, and cross‑modal supervision.

Recent diffusion models have also been widely applied to multimodal synthesis and editing. Notable examples such as DALLE‑2 and Imagen are based on diffusion models, which offer stable training objectives and scalability compared to GANs. The paper categorizes and analyzes methods based on conditional diffusion and pretrained diffusion models.

Autoregressive models handle multimodal data more naturally by first learning a vector‑quantized encoder that discretizes images into token sequences, then modeling token distributions autoregressively. Since text, audio, and other modalities can also be tokenized, these methods unify various multimodal synthesis and editing tasks under a single framework.

While the aforementioned methods focus on 2D image synthesis, the rapid development of NeRF has attracted increasing attention for 3D‑aware multimodal synthesis and editing, which requires multi‑view consistency and presents greater challenges. The paper categorizes and summarizes single‑scene optimized NeRF and generative NeRF approaches.

The fourth chapter compiles popular datasets and modality annotations in the field, and provides quantitative comparisons for typical tasks such as semantic image synthesis, text‑to‑image synthesis, and audio‑guided image editing, along with visualizations of multimodal control results.

The fifth chapter discusses current challenges and future directions, including the need for large‑scale multimodal datasets, accurate and reliable evaluation metrics, efficient network architectures, and further development of 3D perception.

Chapters six and seven address the potential societal impact of this research area and summarize the contributions of the survey.

Readers interested in this review are invited to read the original paper linked at the end of the article.