Why Longer Captions Don't Boost Text‑to‑Image Models: Insights from ByteDance Seed

Text‑to‑image diffusion models have traditionally been scaled by increasing model size, data volume, and compute, while the role of the caption as the primary textual interface for visual supervision has been under‑examined.

The authors ask whether adding more image information via captions can improve learning without changing the backbone. Experiments on several open‑source diffusion systems reveal that caption length quickly saturates: longer natural‑language prompts often yield no gain or even degrade performance, and training on uniformly long captions provides limited benefit.

To investigate, the team conducts a fixed‑backbone image reconstruction study. Starting from a fully annotated image, they generate four captions of increasing detail, keeping the Qwen‑Image model and random seed constant. Despite substantial length growth, reconstruction quality plateaus because the added prose mainly rewrites or connects existing content without introducing new, image‑grounded variables.

Recognizing that token count is a weak proxy, they introduce two complementary metrics to quantify the usable image information in a caption:

Grounded Perplexity Gain (GPG) : a white‑box measure that compares token log‑likelihoods of a frozen vision‑language model with and without the paired image, capturing how much the image makes the caption more predictable.

Effective Detailness (ED) : a black‑box semantic metric that extracts attribute triples from both image and caption, then computes a precision‑weighted F0.5 score, penalizing caption statements lacking visual evidence.

Both metrics correlate strongly with the final diffusion loss: Pearson r = -0.984 for GPG (linear) and r = -0.971 (power‑law) for ED, and their rankings are highly consistent (Spearman ρ = 0.96).

Guided by these findings, the authors propose Structured Prompt (SP) , a JSON‑encoded representation that organizes visual variables into three layers:

Global layer – intent, scene, style, lighting, photography details.

Element layer – identity, attributes, actions, positions, depth, local photography.

Relation layer – spatial, occlusion, interaction, semantic relationships.

SP increases both Diffusability (the model’s ability to learn from the caption) and, when combined with a capable LLM prompter, Promptability (the ability to generate high‑quality SP from a user request). The authors define Generation Quality = Diffusability × Promptability, emphasizing that the product, not a literal multiplication, captures the joint effect.

To build SP at scale, they assemble an image‑to‑SP pipeline: a general‑purpose VLM extracts global semantics, Sapiens adds human pose, DepthAnything V2 provides relative depth, SAM 2.1 supplies masks and occlusion cues, and a final VLM consolidates everything into a consistent JSON prompt.

Training the LLM prompter follows a three‑stage regimen:

Supervised Fine‑Tuning (SFT) on ground‑truth SP distributions.

Cold‑start distillation from privileged reasoning traces that include the paired image.

Reinforcement‑style Fine‑Tuning (RFT) where the prompter’s own rollouts are filtered by a verifier and refined via on‑policy self‑distillation from an image‑conditioned teacher.

Ablations show SFT contributes the largest single‑stage gain, cold‑start improves the mapping from request to SP, and verifier‑gated RFT yields the strongest final performance.

Because SP fields are explicit, errors in generated images can be localized and corrected by editing the relevant JSON entries rather than rewriting an entire natural‑language prompt. The authors implement a refine‑render‑judge loop where the prompter iteratively updates SP, the diffuser renders, and an online judge provides PASS/FAIL feedback with specific issue tags. Experiments indicate that a modest number of refinement rounds (average 2.31 out of a possible 8) suffices, as additional rounds bring diminishing returns.

When evaluated on the same Qwen‑Image backbone, training with Structured Prompt consistently outperforms baseline systems across almost all metrics, especially on compositional, reasoning, and world‑knowledge tasks. Matched‑control experiments confirm that the gains stem from the structured caption interface rather than extra training steps or larger backbones.

Overall, the work demonstrates that scaling text‑conditioning—measured via GPG and ED—and providing a structured, model‑friendly caption format are crucial next steps for advancing text‑to‑image generation beyond merely enlarging diffusion models.