How ImageRAG Boosts Text‑to‑Image Generation with Retrieval‑Augmented Generation

Introduction

Diffusion models have revolutionized image synthesis but struggle with rare or unseen concepts due to limited training data. To address this, the authors propose ImageRAG, a framework that combines Retrieval‑Augmented Generation (RAG) with existing image‑conditioned models, dynamically retrieving relevant images based on a text prompt and using them as contextual guidance without additional RAG‑specific training.

Methodology

The process consists of three steps:

Identify missing concepts: A text‑to‑image (T2I) model first generates an initial image. A visual language model (VLM) then checks whether the image matches the prompt. If not, the VLM lists missing concepts and produces detailed descriptions for each.

Retrieve supporting images: For each description, the system computes CLIP ViT‑B/32 cosine similarity against a large image pool (a subset of LAION) and retrieves the top‑k most similar images. When the pool lacks exact matches, re‑ranking with CLIP, SigLIP, or GPT‑generated captions is explored, though CLIP similarity alone suffices for most experiments.

Incorporate retrieved images: The retrieved images are fed to the downstream model as additional context. For models with in‑context learning (e.g., OmniGen), the prompt is rewritten to include image examples. For encoder‑only models (e.g., SDXL with an IP‑adapter), the images are concatenated as conditioning inputs.

The authors also describe error handling (retry up to three times with higher temperature) and a fallback to direct prompt‑based retrieval when the VLM cannot identify missing concepts.

Implementation Details

ImageRAG uses a LAION subset of 350 k images, CLIP ViT‑B/32 embeddings for similarity, GPT‑4o (2024‑08‑06) as the VLM (temperature 0), OmniGen (Xiao et al., 2024) with a three‑image context window, and SDXL (Podell et al., 2024) equipped with a ViT‑H IP‑adapter (scale 0.5). OmniGen receives up to three concepts with three images each; SDXL receives one concept with one image due to adapter limits.

Experiments and Results

Quantitative evaluation compares OmniGen and SDXL with and without ImageRAG, as well as baselines FLUX (2023), PixArt‑α (2025), and GraPE (2024). Images are generated for ImageNet, iNaturalist (first 1 000 classes), CUB, and Aircraft datasets. CLIP, SigLIP, and DINO similarity scores show consistent improvements for both models when ImageRAG is applied (see Table 1 in the paper).

When proprietary image collections replace the generic LAION pool, performance improves further (Tables 2 and 5), confirming the benefit of domain‑specific retrieval.

Ablation Study

Four ablations are reported:

Re‑phrasing prompts without images yields negligible gains.

Retrieving based on detailed image descriptions outperforms retrieving only missing concepts or the raw prompt.

Increasing the retrieval pool size (1 k → 350 k) generally raises CLIP scores, though diminishing returns appear for stronger base models.

Alternative similarity metrics (SigLIP, GPT‑generated caption re‑ranking, BM25) provide modest gains over plain CLIP similarity, but the added complexity is not justified.

Qualitative Comparison & User Study

Human evaluation with 46 participants (767 pairwise comparisons) shows a clear preference for ImageRAG‑enhanced outputs over baselines and over other RAG‑based generators such as RDM, KNN‑Diffusion, and ReImagen. Participants rated ImageRAG higher on prompt alignment, visual quality, and overall preference (see Figure 6).

Conclusion

ImageRAG demonstrates that a simple, model‑agnostic retrieval pipeline can substantially extend the capability of pretrained T2I systems to generate rare and fine‑grained concepts. By leveraging a VLM to pinpoint missing visual cues and CLIP‑based similarity to fetch supporting images, the approach requires minimal modifications to existing models while delivering consistent quantitative and qualitative gains.