How Should World Models Be Evaluated? Insights from Nanjing University’s Position Paper

Why Evaluating World Models Matters

Recent advances in robotics, autonomous driving, video generation, and embodied AI have popularized the term “world model.” As the concept spreads, a fundamental question arises: what exactly are we measuring when we label a model a world model?

Decision‑Centric Evaluation Focus

A team from Nanjing University’s AI Institute released a position paper that proposes, for embodied decision‑making, evaluation should concentrate on three core abilities: predicting the consequences of actions, judging the quality of different strategies, and providing concrete support for planning and optimization. Metrics such as visual realism, smoothness, and semantic alignment remain useful for diagnosis but are considered secondary.

Broadening Scope of World Models

Two years ago, “world model” primarily referred to environment dynamics used for control and planning. Today the term covers a wide range of technologies, including video generation, interactive neural simulators, latent‑space prediction, data‑synthesis engines, and executable planners. This diversification has turned the term into a broad label, creating varied evaluation needs.

Capability Claims vs. Evaluation Evidence

The paper isolates six typical capability claims found in the literature: (1) predicting future observations, (2) evaluating strategy performance, (3) supporting strategy optimization, (4) generating executable plans, (5) producing valuable training data, and (6) offering implicit representations for decision‑making. It stresses that evaluation evidence must align with the specific claim being made; a model that excels at generating realistic videos does not automatically prove its usefulness for decision‑making.

Current Evaluation Practices

Surveying recent work, the authors find that most evaluations focus on perception‑level metrics: video realism, pixel‑ or perceptual‑level similarity to future trajectories, language instruction alignment, physical consistency, and final task success rate. While these metrics reveal generation quality and basic task performance, they are insufficient for embodied decision‑making, which requires answers to questions such as how a changed action alters outcomes, how reliable reward predictions are, and whether strategy rankings match reality.

Seven‑Level Evaluation Ladder (L0–L7)

L0: Visual Plausibility – Does the generated output look realistic?

L1: Recorded Future Prediction – Can the model forecast future trajectory segments?

L2: Semantic Alignment – Does the output obey the given instruction, task, and scene semantics?

L3: Physical Plausibility – Are basic physical and geometric constraints satisfied?

L4: Action Controllability & Intervention Fidelity – Does changing an action produce the correct task‑related change?

L5: Reward, Value & Outcome Fidelity – Can the model accurately predict success rates, rewards, and progress?

L6: Strategy Evaluation & Ranking – Does the model’s assessment of strategy quality match the real environment?

L7: Planning & Optimization Utility – When placed inside a planner or RL loop, does the model genuinely improve decision quality?

Levels L4–L7 directly address the evidence needed for real‑world embodied systems such as robots and autonomous vehicles.

Decision‑Centric Evaluation Protocol

The authors recommend first declaring a model’s “decision contract” – the task family, strategy type, action interface, time horizon, and whether the model is used for prediction, evaluation, planning, or optimization. With this contract, targeted evaluations can be performed:

Intervention Fidelity Test: Keep the history fixed, alter the action branch, and check whether the model produces the correct task‑related change.

Closed‑Loop Rollout: Run a policy inside the model and compare the closed‑loop behavior with the real environment.

Reward & Success Calibration: Measure how trustworthy the model’s predictions of reward, progress, and success probability are.

Strategy Ranking Consistency: Compare the ordering of strategies produced by the model with the ordering observed in the real world.

Optimization Gain Assessment: With a fixed optimization budget, verify that the model yields reproducible performance improvements.

Exploitability & Uncertainty Measurement: Test whether the model can be easily fooled by an optimizer into high‑valued but unrealistic trajectories and whether its uncertainty estimates aid risk control.

Practical Minimal‑Viable Reporting for Real Robots

Recognizing the high cost of robot experiments, the paper proposes a minimal reporting scheme that includes three evidence types: (1) a small set of reset‑matched action‑branch experiments to verify intervention effects, (2) several clearly strong and weak fixed strategies for success‑rate calibration and ranking tests, and (3) execution verification of high‑scoring trajectories generated by the model.

Overall Contributions

In summary, the paper makes three key contributions: (1) a systematic taxonomy of what current world‑model literature actually measures, (2) an analysis of how different evidence supports distinct capability claims, and (3) a concrete, decision‑oriented evaluation framework that turns abstract questions about embodied intelligence into comparable, reportable metrics.