How Should World Models Be Evaluated? Insights from Nanjing University’s Position Paper

Recent discussions have amplified the term "world model" across robotics, autonomous driving, video generation, and embodied AI. The Nanjing University AI Institute team released a position paper titled "How Should World Models Be Evaluated for Embodied Decision-Making? A Decision-Making-Centric Position" that asks what we truly evaluate when a model is called a "world model".

The paper argues that for embodied decision‑making, evaluation should prioritize the model's ability to predict action consequences, judge strategy quality, and support planning or optimization. Traditional video realism metrics remain useful as diagnostics but are secondary to decision‑oriented evidence.

Scope of "World Model"

Two years ago, world models mainly referred to environment models for control and planning, where a system internally simulates "if these actions are taken, what will happen" to aid policy evaluation and path planning. Today the term covers many objects: dynamics conditioned on actions, future video generation, interactive neural simulators, latent‑space representation prediction, data synthesis engines, and executable planners. This diversification widens research but also blurs definitions, demanding clearer evaluation criteria.

Core Capability Claims

Predict future observations.

Evaluate different strategies.

Support strategy optimization.

Generate executable plans.

Produce valuable training data.

Provide implicit representations that support decisions.

These claims differ; video realism is not equivalent to decision utility. The paper stresses aligning evaluation evidence with the claimed capability.

Current Evaluation Practice

A survey of recent works shows most metrics target:

Visual realism and aesthetics of generated video.

Pixel‑level or perceptual similarity to real future trajectories.

Understanding of language instructions and semantic consistency.

Surface adherence to physical laws.

Overall task success rate.

While useful for diagnosing generation quality, these metrics alone cannot fully support embodied decision claims.

7‑Level Evaluation Ladder (L0–L7)

L0: Visual plausibility – does the output look realistic?

L1: Recorded future prediction – can the model forecast future trajectory segments?

L2: Semantic alignment – does the output match instructions, tasks, and scene semantics?

L3: Physical plausibility – does it satisfy basic physics and geometry?

L4: Action controllability & fidelity – does changing an action produce the correct task‑relevant change?

L5: Reward, value & outcome fidelity – can the model accurately predict success, reward, and progress?

L6: Strategy evaluation & ranking – does the model’s ranking of candidate strategies match the real environment?

L7: Planning & optimization utility – when placed in a planner or RL loop, does the model genuinely improve decision quality?

Lower levels serve as diagnostics; higher levels directly correspond to decision‑making value, especially for robotics, autonomous driving, and agent planning.

Decision‑Centric Evaluation Protocol

Researchers should first declare a "decision contract": task family, strategy type, action interface, time horizon, and whether the model is used for prediction, evaluation, planning, or optimization. Then the evaluation should include:

Intervention action fidelity test – keep history fixed, change the action branch, and verify the correct task‑related outcome.

Closed‑loop policy rollout – run policies inside the model and compare behavior to the real environment.

Reward & success‑rate calibration – assess the trustworthiness of predicted reward, progress, and success probability.

Strategy ranking consistency – compare model‑internal strategy ordering with real‑world ordering.

Optimization gain assessment – with a fixed optimization budget, check whether the model yields reproducible performance gains.

Exploitable region & uncertainty testing – measure whether optimizers can find over‑optimistic regions and whether uncertainty estimates aid risk control.

Minimal Viable Reporting for Real‑World Systems

Because large‑scale robot experiments are costly, the authors suggest a minimal set of three evidence types:

Few reset‑matched action‑branch experiments to test outcome changes.

Several clearly strong vs. weak fixed strategies to test success‑rate calibration and ranking.

Execution verification of high‑scoring trajectories generated by the model.

This keeps evaluation feasible while moving from "looks good" to "decision‑reliable".

Takeaways

The paper accomplishes three things: (1) systematically maps what current world‑model literature actually measures; (2) clarifies which evidence supports which capability claims; (3) provides an executable evaluation framework that turns decision‑relevant questions into comparable metrics. It urges the community to shift focus from visual quality toward action consequences, long‑term returns, and strategy ordering for truly embodied AI.