Contextual Learning for Personalized Text‑to‑Image Generation

Background: Text‑to‑image generation models such as Imagen, Stable Diffusion, DALL·E 2 and Midjourney can produce high‑quality images from prompts, but they rely solely on text as the control signal, which limits precise specification of object position, angle, or personal subjects.

Motivation: To enable personalized generation without costly model fine‑tuning, the talk proposes using in‑context learning for image generation, inspired by the success of in‑context learning in large language models.

Design – Architecture: The proposed network reuses the UNet encoder of diffusion models, adding an extra attention layer that can attend to multiple example image‑text pairs (neighbors). These examples are encoded into feature maps and concatenated with the noisy target image during denoising.

Design – Training data: A large ICL‑style dataset is built by clustering web‑scraped image‑text pairs, cleaning text with large language models, and augmenting with synthetic examples generated by Dream Booth. Filtering based on CLIP similarity retains high‑quality clusters for training.

Training procedure: The SuTI (Subject‑Driven Text‑to‑Image) model is trained on ~500 K examples for one day, using the concatenated neighbor features and text prompts to guide diffusion.

Results and Outlook: SuTI demonstrates strong stylization, viewpoint, attribute, and accessory control, achieving high alignment scores and photorealism comparable to Dream Booth while being faster. Future work includes scaling the model, adding pose control, and releasing it on Google Cloud.

Q&A highlights: The audience asked about how style and angle are learned, the role of data, extending skills via data, and integrating other encoders or ControlNet‑style signals.