Reference Image Generation for Subject‑Driven Diffusion

Current text‑to‑image models (e.g., Flux, Stable Diffusion 3.5, Midjourney) can generate photorealistic images, yet their ability to control output via text alone is limited, preventing tasks such as generating a “GitHub‑style” marketing image.

Generating high‑quality product images in diverse scenes is crucial for both B2B and B2C content creation, motivating research into more controllable generation methods.

Two main paradigms exist: (1) inpainting, which redraws background while preserving the target object but struggles to blend objects into new contexts; (2) reference‑image generation, also known as subject‑driven or personalized image generation.

Reference‑image generation can be performed with test‑time fine‑tuning (e.g., DreamBooth, Text Inversion, Custom Diffusion) or without additional tuning (zero‑shot). DreamBooth fine‑tunes a diffusion model on 3‑5 images of a specific subject, using a rare token and a class‑preservation loss to maintain diversity.

While test‑time fine‑tuning yields high consistency, it requires per‑object model updates, which is operationally costly.

Zero‑shot approaches such as IP‑Adapter and Animate‑anyone (ReferenceNet) inject image features into the diffusion process. IP‑Adapter uses a CLIP image encoder but provides limited fine‑grained detail. ReferenceNet adds a multi‑scale feature branch, improving detail preservation and consistency, and has been applied to video synthesis and virtual try‑on.

Data for reference‑image generation are collected either as reconstruction sets (object masks with augmentation) or paired sets (same object in different scenes). Experiments showed that paired data improve diversity.

Model backbones were evaluated: SD‑1.5 lacked sufficient detail rendering, so SD‑XL and Flux were adopted for their superior text and texture fidelity.

The final pipeline combines a virtual‑try‑on framework with ReferenceNet‑style feature injection, achieving fine‑grained product consistency across scenes.

Quantitative and qualitative results demonstrate that the proposed method outperforms existing SOTA solutions in consistency while preserving shape, texture, and text.

Limitations include insufficient handling of diverse product categories, image aesthetics, and human subjects, as well as occasional incorrect object‑person interactions.

Future work will expand category coverage, improve data quality, and enhance model robustness.

References include recent works on DreamBooth, Textual Inversion, IP‑Adapter, Animate‑anyone, and ControlNet.