How Chain‑of‑Thought Boosts Text‑to‑Image Generation: The New o1 Inference Scheme

Overview

Large language models (LLMs) and large multimodal models (LMMs) have achieved strong results on text, 2‑D images, video and 3‑D point clouds. Recent work such as OpenAI o1 demonstrates that enhancing chain‑of‑thought (CoT) reasoning can boost performance on math, science and coding tasks.

Problem Statement

While CoT reasoning is widely used for complex understanding, its applicability to verifying and strengthening autoregressive image‑generation pipelines has not been studied. The goal is to determine whether test‑time verification, preference alignment, and their integration can improve generation quality.

Methods and Models

Three techniques are explored:

Extended test‑time computation using reward models as validators.

Direct Preference Optimization (DPO) to align model preferences with human judgments.

Integration of the two for complementary effects.

Reward Model Taxonomy

Two categories of reward models are considered:

Outcome Reward Model (ORM) : rescoring complete inference outputs and selecting the most confident candidate.

Process Reward Model (PRM) : providing a reward score for every candidate at each generation step.

ORM Implementation

Three variants are built:

Zero‑shot ORM : a pre‑trained LLaVA‑OneVision (7B) model receives the text prompt and generated image, and a crafted prompt template (see Appendix) elicits a binary response (“yes” for high quality, “no” for low quality). The candidate with the highest “yes” probability is chosen.

Ranking data construction : GPT‑4 generates a list of 200 everyday objects with specific colors. Six GenEval object‑centric prompt templates turn the list into diverse text prompts. For each prompt, the baseline Show‑o model generates ~50 images at high temperature. GenEval metrics label each image “yes”/“no”, forming a ranking dataset.

Fine‑tuned ORM : the ranking dataset fine‑tunes LLaVA‑OneVision (batch size 8, learning rate unspecified). The fine‑tuned model captures finer visual‑text relationships and yields more reliable scores.

PRM Implementation

A zero‑shot PRM based on LLaVA‑OneVision is first built. A 10 K‑step text‑to‑image ranking dataset is curated (details in Appendix) and used to fine‑tune the PRM, enabling step‑wise reward estimation.

Potential‑Assessment Reward Model (PARM)

PARM combines ORM and PRM strengths through three progressive tasks:

Clarity judgment : for each sampled inference path, PARM checks whether the partially generated image is visually clear enough for evaluation, assigning a binary label. Paths labeled “no” are discarded.

Potential assessment : for “yes” paths, PARM predicts whether the current step can lead to a high‑quality final image, again using a binary label. “No” paths are truncated.

Best‑choice selection : remaining high‑potential paths are ranked with an ORM‑style global selector; the path with the lowest “no” probability is output.

Test‑time Preference Alignment with DPO

DPO is applied directly to the autoregressive image generator. The policy (initialized from Show‑o) is optimized while a reference policy (also initialized from Show‑o) remains fixed, encouraging higher likelihood for preferred images. Training runs for one epoch with batch size 10 and learning rate unspecified.

Iterative DPO

After the first DPO round, the model generates new ranked data using updated Chinese prompts. Images are labeled “yes”/“no” with the same ORM procedure; pairs with identical labels are removed, yielding a refined 7 K‑sample DPO ranking set. A second DPO pass further improves preference modeling.

Experiments and Results

All reward models are evaluated on the GenEval benchmark. Key findings:

PARM outperforms other reward models as a test‑time validator, improving ORM‑based scores by 6 % and scaling effectively with increased test‑time computation.

PARM surpasses iterative DPO, demonstrating superior utilization of post‑training complementary advantages and achieving higher integrated scores than fine‑tuned ORM.

When PARM is combined with iterative DPO, the Show‑o model reaches the best configuration, raising the overall GenEval score by an unspecified margin and surpassing Stable Diffusion 3 by 15 %. Notable gains appear in compositional scenarios such as “two objects”, “color”, “position”, and “attribute binding”.

Conclusion

The systematic study shows that extending test‑time computation, aligning preferences with DPO, and integrating both can substantially improve autoregressive image generation. The introduced PARM and its enhanced version PARM++ assess step‑wise generation potential and incorporate a reflection mechanism for self‑correction, highlighting a promising direction for CoT‑enhanced image synthesis.