ImageRAG: Leveraging RAG and AIGC to Elevate Image Generation Quality

Problem

Diffusion‑based text‑to‑image (T2I) models generate high‑quality visuals but often fail on rare or unseen concepts because the training data do not contain enough examples of those concepts.

Method Overview

The ImageRAG pipeline proceeds in four stages:

Generate an initial image from the user prompt using a pretrained T2I model.

Feed the prompt and the image to a visual language model (VLM). The VLM answers whether the image matches the prompt.

If the answer is negative, the VLM enumerates the missing concepts and, for each concept, produces a detailed textual description that can be used for retrieval.

Retrieve visually similar images from an external corpus with CLIP‑based cosine similarity and inject the retrieved images as additional context for a second generation step.

Choosing Reference Images

Only concepts that the base model cannot render are retrieved, limiting the number of reference images. When the VLM cannot name a missing concept, it is queried up to three times with an increased temperature to encourage more diverse vocabularies.

Retrieval Process

All images in the retrieval corpus are indexed with CLIP ViT‑B/32 embeddings. For each description the pipeline computes cosine similarity to every corpus embedding and selects the top‑k most similar images. Experiments (see Table 4) show that plain CLIP similarity is sufficient; more elaborate re‑ranking (e.g., SigLIP, GPT‑generated captions, BM25) yields only marginal gains.

Using Retrieved Images

Two families of T2I models are supported:

In‑context learning (ICL) models such as OmniGen (Xiao et al., 2024) that accept multiple reference images as examples.

Standard diffusion models enhanced with an image encoder , e.g., SDXL (Podell et al., 2024) equipped with the ViT‑H IP‑adapter (Ye et al., 2023).

For each missing concept a prompt template is built, e.g., Based on these examples, generate <concept>, and the retrieved images are inserted as visual examples.

Implementation Details

The retrieval set is a subset of LAION (Schuhmann et al., 2022) containing up to 350 k images. CLIP ViT‑B/32 provides the similarity metric. The VLM is GPT‑4o (2024‑08‑06) with temperature 0 (unless a retry is needed). Base generators:

OmniGen (Xiao et al., 2024) – default parameters, image‑guidance 1.6, classifier‑free guidance 2.5, resolution 1024×1024, supports up to three reference images per prompt (one per missing concept).

SDXL with the ViT‑H IP‑adapter (Ye et al., 2023) – ip_adapter_scale=0.5, supports a single reference image per prompt.

Experiments

Quantitative Comparison

ImageRAG was evaluated on four benchmarks: ImageNet (Deng et al., 2009), iNaturalist (first 1 000 classes, Van Horn et al., 2018), CUB (Wah et al., 2011) and Aircraft (Maji et al., 2013). For each dataset the CLIP (Radford et al., 2021), SigLIP (Zhai et al., 2023) and DINO (Zhang et al.) similarity scores were computed. Both OmniGen and SDXL show consistent improvements when ImageRAG is applied, outperforming strong baselines FLUX (2023), PixArt‑α (2025) and GraPE (2024).

Proprietary‑Data Generation

When the retrieval corpus is restricted to a domain‑specific subset (e.g., a private brand collection) rather than the full LAION pool, ImageRAG yields additional gains (see Tables 2 and 5), demonstrating usefulness for personalized or brand‑specific generation.

Ablation Studies

Re‑phrasing the prompt without providing reference images does not close the performance gap.

Retrieving with detailed image descriptions outperforms retrieval using only the missing concept name or the raw prompt.

Increasing the corpus size from 1 k → 10 k → 100 k → 350 k examples improves CLIP scores for both models; however, even a modest 1 k‑image set already surpasses the baseline.

Alternative similarity metrics (SigLIP, GPT‑generated description re‑ranking, BM25) give marginal improvements over plain CLIP cosine similarity, which does not justify the added complexity.

User Study

46 participants performed 767 pairwise comparisons. Participants consistently preferred ImageRAG‑enhanced outputs over the baselines and over other retrieval‑based methods (RDM [Blattmann et al., 2022], KNN‑Diffusion [Shen et al., 2022], ReImagen [Chen et al., 2022]) across three criteria: textual alignment, visual quality, and overall preference.

Conclusion

ImageRAG demonstrates that a simple, model‑agnostic retrieval‑augmented generation loop—driven by VLM‑based concept detection and CLIP similarity search—significantly improves diffusion models’ ability to render rare and fine‑grained concepts. The approach requires only minimal changes to existing T2I pipelines and works with both in‑context‑learning models and diffusion models equipped with image adapters.