PromptEcho: Leveraging Frozen Multimodal Models for High‑Quality Text‑to‑Image Rewards Without Labels

Problem

Reinforcement learning for text‑to‑image models needs a reward that accurately reflects prompt following. Conventional metrics such as CLIP Score are too coarse to capture attributes, spatial relations, or counting. Existing open‑source reward models (PickScore, ImageReward, HPS v2) are limited by model size and scarce annotations, and training a high‑quality reward model is costly.

PromptEcho Method

PromptEcho leverages a frozen vision‑language model (VLM) to obtain a reward without any additional training. For each generated image, a fixed query (e.g., "Describe this image in detail") and the original prompt are fed to the VLM in teacher‑forcing mode, forcing the model to predict each token of the prompt. The negative token‑level cross‑entropy is used as the reward:

reward = -\sum_{t}\log P_{VLM}(prompt_t \mid image, query)

The reward equals the log‑likelihood that the VLM can "echo" the prompt after seeing the image. A correct image (e.g., a red cat on a blue table) yields a high likelihood; deviations lower the likelihood and thus the reward.

Why not direct VLM scoring?

Directly asking the VLM to output a discrete score (method named InferScore) suffers from hallucination, sampling noise, and, importantly, cannot distinguish subtle differences between images generated from the same prompt—often assigning identical scores. PromptEcho’s continuous likelihood is deterministic and fine‑grained.

Experiments

Training data construction . Approximately 100 k high‑quality images were collected. Using Qwen3‑VL‑32B with the query "Describe this image in detail", dense captions of 200–400 words (covering objects, attributes, spatial relations, colors, textures) were generated. These captions formed the RL prompt set.

DenseAlignBench . A held‑out test set of 2 000 captions not present in the training data was built. Evaluation used Gemini‑3‑flash‑preview pairwise scoring to measure prompt‑following ability.

PromptEcho was applied to two state‑of‑the‑art open‑source text‑to‑image models (Z‑Image and QwenImage‑2512) with Qwen3‑VL‑32B as the reward VLM. Across all public benchmarks, PromptEcho consistently improved instruction‑following performance, demonstrating that the reward originates from the VLM’s pre‑training knowledge and generalizes across distributions and model architectures.

Scaling of Reward VLM

Replacing the 32‑billion‑parameter Qwen3‑VL‑32B with its 8‑billion‑parameter counterpart reduced performance on every key metric, confirming that larger VLMs provide higher‑quality reward signals and that PromptEcho scales with VLM size.

PromptEcho vs. InferScore

When both methods used Qwen3‑VL‑32B, PromptEcho markedly outperformed InferScore on DenseAlignBench; InferScore even fell below the baseline, validating the superiority of continuous likelihood rewards.

Generalization to Text Rendering

The same mechanism was transferred to an e‑commerce poster text‑rendering task. The query changed to a structured OCR prompt, and the label format switched from dense captions to JSON‑encoded text tags. After RL, the overall text‑correctness rate on 5 000 test samples rose from 68 % to 75 % (+7 percentage points), showing that the reward paradigm adapts to different visual generation objectives without retraining a dedicated reward model.

Case Study

Visual comparisons between the baseline QwenImage‑2512 and the PromptEcho‑fine‑tuned model show notable improvements in detail fidelity, spatial relationships, and object counting, matching the quantitative gains reported in the benchmarks.

Conclusion

PromptEcho demonstrates that the pre‑training loss of a frozen VLM itself serves as a high‑quality image‑text alignment reward, eliminating the need for annotation or reward‑model training. As open‑source VLMs continue to improve, the reward quality and downstream optimization benefits are expected to increase.

Code, model weights, and the DenseAlignBench dataset are publicly released at https://github.com/roooobotx/prompt_echo.