TerraVis: Evaluating World Consistency in Text-to-Image Generation

AI-generated images can score highly on aesthetics, prompt alignment, and human preference metrics while still containing distorted limbs, interpenetrating objects, or physically impossible phenomena. These errors often go undetected by existing evaluation metrics, revealing a blind spot: whether objects, interactions, and spatial relationships in the image conform to real-world constraints.

TerraVis Framework

Accepted at NeurIPS 2026 Evaluations and Datasets Track, TerraVis (Towards Evaluation of World-Grounded Visual Consistency in Text-to-Image Generation via MLLM Workflows) proposes a "world consistency" evaluation framework. It assesses images without relying on the generation prompt, focusing on macroscopic objects, interactions, and scenes across structural, spatial, physical, biological, and commonsense dimensions.

World consistency is distinct from prompt alignment. A desk image scored 1.00 on TIFA alignment but received a human world consistency score of 0 due to a structurally distorted laptop. Conversely, an image violating a "no water" fountain prompt received the highest world consistency score because the fountain itself behaved realistically. Prompts may also intentionally request unrealistic scenes (e.g., floating books), where low world consistency reflects surreal intent, not failure. Stylized images (cartoons, illustrations, sketches) are evaluable as long as they contain recognizable macroscopic objects; pure abstract patterns, icons, charts, and UI screenshots return N/A.

Three-Level, 18-Category Taxonomy

TerraVis decomposes "looks wrong" into 18 checkable issue types across three levels:

Object level (7): identity, structure, anatomy, texture, text, etc.

Interaction level (8): contact, support, motion, optics, etc.

Scene level (3): relative scale, occlusion, region continuity.

Evaluation proceeds in four steps: (1) determine image applicability, (2) detect issues per category, (3) classify as major or minor, (4) compute score via fixed formula. Steps 1–3 are executed by a multimodal LLM; step 4 uses an exponential scoring function: no issues = 1.0, one minor ≈ 0.61, one major ≈ 0.37. Major violations affect the subject, foreground objects, or core events; minor violations affect only background or peripheral elements. Each category reports at most the most salient anomaly with affected objects and visual evidence.

Abstract violation names are rewritten as concrete visual conditions (e.g., "Do two objects that should touch appear to interpenetrate?"), turning holistic judgment into evidence-based itemized checks.

Human Annotation Study

On COCO-T2I and GenAI-Bench, five models (SD 3.5-Large, Qwen-Image, FLUX.2-dev, GPT Image 1.5, Nano Banana Pro) were evaluated. 4,500 images (half per benchmark) were independently scored by three annotators each, yielding 13,500 annotations. The automatic evaluator in main experiments was GPT-5.5.

By majority vote (≥2 annotators flag an issue), 48.6% (COCO-T2I) and 69.7% (GenAI-Bench) of images exhibited world consistency issues. The higher GenAI-Bench rate reflects its inclusion of fictional or unrealistic prompts, so this is not a direct model failure rate.

Correlation with Human Judgment

Merging all five models' images, TerraVis achieved Spearman rank correlations of 0.44 (COCO-T2I) and 0.43 (GenAI-Bench) with human world consistency scores — highest among compared metrics. The best existing metric, RAHF, scored 0.29 and 0.37 respectively. GPTScore produced constant output on COCO-T2I, making correlation incalculable.

An ablation variant TerraVis-H, given the same category definitions and human guidelines but producing a holistic score directly, achieved only 0.28 on both benchmarks, confirming that itemized checking outperforms holistic scoring even with identical criteria.

Against human majority-vote labels, TerraVis recall improved from TerraVis-H's 15.4%/19.8% to 51.9%/48.3%, at the cost of precision (75.8%/89.1%), indicating more false positives.

Ablation Studies

Removing any of the three levels reduced correlation, with the object level contributing most. Removing individual categories also lowered correlation; structural distortion had the largest single impact.

The pipeline generalizes across styles and evaluator models. On stylized images correlation was 0.54 vs. 0.41 on photorealistic (limited stylized samples prevent claiming superiority). Replacing GPT-5.5 with open-source Gemma 4 31B yielded correlations of 0.42 and 0.51, demonstrating feasibility of open evaluators.

Model Rankings Shift

World consistency reorders model rankings. On GenAI-Bench, GPT Image 1.5 led on traditional metrics, but both TerraVis and human world consistency scores ranked FLUX.2-dev first. Qwen-Image topped aesthetics but placed fourth on human world consistency. FLUX.2-dev tends to generate simpler scenes, which may reduce violation opportunities, though this remains a qualitative observation.

Violation composition analysis shows text/symbol issues rank top-three for every model on both benchmarks. Object-level issues dominate, but interaction and scene levels combined account for roughly one-quarter to one-half of detected violations, meaning object-only checks miss substantial problems.

Limitations and Failure Cases

TerraVis still makes errors: it catches garbled book text humans miss, but misjudges cat anatomy or misses dense-scene child arm distortions. On a restaurant flamingo image, three annotators gave 3, 5, and 1 (1–5 scale), showing even humans disagree on commonsense boundaries. The multi-stage pipeline requires more inference time, detection has false positives/negatives, and the 18 categories will need ongoing revision as generative models evolve.

Availability

TerraVis provides an online evaluation demo at https://shyfoo.github.io/TerraVis-web/ where users can upload images (no prompt required) and view per-category progress, final score, issue types, visual evidence, and major/minor classifications. Pre-computed examples are also available. Code: https://github.com/ShyFoo/TerraVis; Dataset: https://huggingface.co/datasets/ShyFoo/TerraVis-Annotations; Paper: https://arxiv.org/abs/2610.02959.

TerraVis makes world consistency a quantifiable evaluation target, linking scores to specific visual problems to help researchers compare models, analyze failure types, and direct improvements. As generated images become increasingly photorealistic, world consistency should join aesthetics, alignment, and preference as a core standard for measuring text-to-image progress.