Skip to content
Asking the World: AI Builds Physical Simulations from Video to Answer Counterfactual Questions

Asking the World: AI Builds Physical Simulations from Video to Answer Counterfactual Questions

Tsinghua's Asking the World (ATW) framework enables AI to autonomously build executable physical simulations from video, then use queries, simulations, and interventions to answer physical property, prediction, and counterfactual reasoning questions, achieving 80.82% on CLEVRER and 70.56% on ContPhy benchmarks with Gemini-3-Flash.

Machine Heart
Machine Heart
Machine Heart

From Video to Executable Physical Worlds

Tsinghua University's Asking the World (ATW) framework lets an AI agent construct a runnable physical simulation from a video and then answer physical reasoning questions by querying parameters, continuing the simulation, or intervening in the scene. Building on the team's earlier PhysMind work, ATW recovers geometry, motion, and physical parameters (mass, friction, restitution) through a process called "system identification," creating a calibrated world that can be reused for multiple questions.

Figure 1: PhysMind modifies the reconstructed scene and runs simulation to judge which collisions still occur after removing a sphere.
Figure 1: PhysMind modifies the reconstructed scene and runs simulation to judge which collisions still occur after removing a sphere.

For example, to answer "which collisions still happen if a ball is removed?" PhysMind deletes the ball in the reconstructed scene, re‑runs the simulation, and reads the resulting trajectories and collision events.

Figure 2: PhysMind pipeline including geometry and motion recovery, physical parameter estimation, and reasoning via query, simulation, and intervention.
Figure 2: PhysMind pipeline including geometry and motion recovery, physical parameter estimation, and reasoning via query, simulation, and intervention.

World Modeling and Probing Phases

ATW organizes the process into two agent‑driven phases: "World Modeling" and "World Probing." In the modeling phase, the agent selects relevant objects and physical models, calls vision tools to recover geometry and motion, fits physical parameters using a continuous‑time analytical method to estimate friction, mass ratios, and restitution coefficients, and iteratively corrects perception or model choices based on intermediate feedback. A billiard example shows the agent detecting a missed yellow ball, adding it, and proceeding.

Figure 3: ATW selects operations based on task and feedback, including correcting missed detections, removing a brown ball, and querying collision events.
Figure 3: ATW selects operations based on task and feedback, including correcting missed detections, removing a brown ball, and querying collision events.

In the probing phase, the agent chooses the appropriate operation: querying fitted parameters for property questions, continuing simulation for predictions, or copying the scene and applying interventions for counterfactuals. If evidence is insufficient, it runs additional queries or experiments. All experiments share the same calibrated world, enabling direct comparison of outcomes.

PolyWorld Engine for Diverse Materials

To go beyond rigid bodies, the team developed the PolyWorld Engine, built on NVIDIA Warp. It supports rigid bodies, soft bodies, cloth, ropes, fluids, and their couplings through a unified interface for scene construction, state reading, simulation stepping, and intervention. The engine aligns simulated motion and deformation with video observations by searching parameters and minimizing differences; for deformable objects it compares shape, surface, or centerline changes over time. If discrepancies stem from object correspondence or physics model choice, the agent can revise the modeling phase and re‑fit.

Figure 4: ATW applies unified modeling and experimental reasoning to rigid, fluid, and cloth scenarios.
Figure 4: ATW applies unified modeling and experimental reasoning to rigid, fluid, and cloth scenarios.

Benchmark Results

ATW is evaluated on two physical reasoning benchmarks using Gemini‑3‑Flash as the base vision‑language model, without additional training or fine‑tuning, offering a lower‑cost alternative to models like Astra:

CLEVRER (rigid‑body collisions): overall accuracy 80.82%, a 46.50‑point gain over direct answering and 8.27 points over PhysMind.

ContPhy (multiple materials, continuous media): overall accuracy 70.56%, a 28.10‑point gain over direct answering.

Table 1: CLEVRER results comparing direct answering, PhysMind, and ATW.
Table 1: CLEVRER results comparing direct answering, PhysMind, and ATW.
Table 2: ContPhy results covering property judgment, prediction, counterfactual, and goal‑driven questions.
Table 2: ContPhy results covering property judgment, prediction, counterfactual, and goal‑driven questions.

Real‑World Evaluation

On 60 real‑world videos (billiards, bouncing balls, toy cars) with 60 questions, ATW achieves 71.67% accuracy versus 33.33% for direct answering, demonstrating potential for broader real‑scene deployment.

Figure 5: ATW object segmentation, motion tracking, and dynamic reconstruction in real scenes.
Figure 5: ATW object segmentation, motion tracking, and dynamic reconstruction in real scenes.

References

Project page: https://asking-the-world.github.io/

Code: https://github.com/zengsx23/Asking-the-World

Paper: https://arxiv.org/abs/2609.39135

PhysMind project page: https://physmind.github.io/

PhysMind code: https://github.com/ccyydd/PhysMind

PhysMind paper: https://arxiv.org/abs/2608.04575

Original Source

Signed-in readers can open the original source through BestHub's protected redirect.

Sign in to view source
Republication Notice

This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactadmin@besthub.devand we will review it promptly.

Simulationvideo understandingcounterfactual reasoningWarpGemini-3-Flashphysical reasoningAsking the WorldATWCLEVRERContPhyPolyWorld Enginesystem identification
Reader feedback

How this landed with the community

Sign in to like

Rate this article

Was this worth your time?

Sign in to rate
Discussion

0 Comments

Thoughtful readers leave field notes, pushback, and hard-won operational detail here.