Why High‑Quality Video Isn’t Enough: Inside the WorldArena Embodied AI Benchmark

Background

High‑resolution video masks the inability of robots to make decisions in the physical world. Traditional evaluation focuses on visual quality, ignoring whether models can guide robot actions. WorldArena is a unified benchmark for evaluating embodied world models (EWMs).

Benchmark Design

WorldArena combines 16 objective video‑quality metrics with three functional modules: open‑loop video generation, closed‑loop embodied tasks, and human subjective evaluation. Scores are normalized to 0‑100 and averaged into the composite EWMScore .

Video‑quality dimensions

Exposure, noise, compression artifacts – measured by MUSIQ.

Aesthetic quality – assessed by the LAION predictor.

Motion plausibility – RAFT optical‑flow model evaluates smoothness and physical feasibility of robot arm movements.

Content consistency – DINO feature similarity across frames.

Background stability – detection of unwanted distortions.

Physical compliance – Qwen3‑VL checks contact dynamics and force realism.

3D accuracy – depth‑map comparison between generated and real videos.

Controllability – alignment between textual commands and generated video content.

Evaluation Procedure

70 human annotators reviewed 3,500 test videos, scoring overall quality, instruction adherence, and physical compliance, and performed paired comparisons to obtain a win‑rate metric. All dimension scores are linearly normalized to a 0‑100 range and averaged to produce the final EWMScore .

Models Evaluated

Fourteen models were benchmarked, including open‑source video generators (e.g., CogVideoX), commercial models (Wan 2.6, Veo 3.1), and embodied‑specific models (Genie Envisioner, CtrlWorld, IRASim, RoboMaster, WoW, etc.). General‑purpose video models excel on aesthetic metrics, while embodied models achieve higher scores on physical interaction and consistency.

Core Tasks

Task 1 – Data‑synthesis engine : Models generate future observation videos for dual‑arm robot tasks from the RoboTwin 2.0 dataset (50 tasks). Synthetic trajectories were used to train downstream policies. Only RoboMaster and WoW surpassed policies trained on real physics data.

Task 2 – Strategy evaluation officer : Models observe control commands and generate corresponding environment videos. CtrlWorld’s outputs correlated strongly (Pearson r ≈ 0.9) with a physical simulator, whereas Cosmos‑Predict 2.5 showed weak correlation and some models over‑fitted to successful outcomes.

Task 3 – End‑to‑end planner : Models receive textual instructions and must output full action plans as videos and joint commands. Performance was poor; all models scored far below a dedicated vision‑language‑action (VLA) controller.

Correlation Analysis

EWMScore correlates with human subjective ratings (Pearson r = 0.825). Correlation with data‑synthesis success is 0.600, and with end‑to‑end action execution is 0.360, indicating that visual fidelity alone does not guarantee embodied intelligence.

Conclusions

Current EWMs can predict short‑term visual futures but struggle with long‑horizon closed‑loop control. High‑quality video generation is necessary but insufficient for real‑world robotic competence. Bridging the gap from visual effects to physics‑aware robot control remains a major challenge.

References

ArXiv paper: https://arxiv.org/pdf/2602.08971

WorldArena website: https://world-arena.ai/

GitHub repository: https://github.com/tsinghua-fib-lab/WorldArena