Asking the World: AI Builds Physical Simulations from Video to Answer Counterfactual Questions
Tsinghua's Asking the World (ATW) framework enables AI to autonomously build executable physical simulations from video, then use queries, simulations, and interventions to answer physical property, prediction, and counterfactual reasoning questions, achieving 80.82% on CLEVRER and 70.56% on ContPhy benchmarks with Gemini-3-Flash.
From Video to Executable Physical Worlds
Tsinghua University's Asking the World (ATW) framework lets an AI agent construct a runnable physical simulation from a video and then answer physical reasoning questions by querying parameters, continuing the simulation, or intervening in the scene. Building on the team's earlier PhysMind work, ATW recovers geometry, motion, and physical parameters (mass, friction, restitution) through a process called "system identification," creating a calibrated world that can be reused for multiple questions.
For example, to answer "which collisions still happen if a ball is removed?" PhysMind deletes the ball in the reconstructed scene, re‑runs the simulation, and reads the resulting trajectories and collision events.
World Modeling and Probing Phases
ATW organizes the process into two agent‑driven phases: "World Modeling" and "World Probing." In the modeling phase, the agent selects relevant objects and physical models, calls vision tools to recover geometry and motion, fits physical parameters using a continuous‑time analytical method to estimate friction, mass ratios, and restitution coefficients, and iteratively corrects perception or model choices based on intermediate feedback. A billiard example shows the agent detecting a missed yellow ball, adding it, and proceeding.
In the probing phase, the agent chooses the appropriate operation: querying fitted parameters for property questions, continuing simulation for predictions, or copying the scene and applying interventions for counterfactuals. If evidence is insufficient, it runs additional queries or experiments. All experiments share the same calibrated world, enabling direct comparison of outcomes.
PolyWorld Engine for Diverse Materials
To go beyond rigid bodies, the team developed the PolyWorld Engine, built on NVIDIA Warp. It supports rigid bodies, soft bodies, cloth, ropes, fluids, and their couplings through a unified interface for scene construction, state reading, simulation stepping, and intervention. The engine aligns simulated motion and deformation with video observations by searching parameters and minimizing differences; for deformable objects it compares shape, surface, or centerline changes over time. If discrepancies stem from object correspondence or physics model choice, the agent can revise the modeling phase and re‑fit.
Benchmark Results
ATW is evaluated on two physical reasoning benchmarks using Gemini‑3‑Flash as the base vision‑language model, without additional training or fine‑tuning, offering a lower‑cost alternative to models like Astra:
CLEVRER (rigid‑body collisions): overall accuracy 80.82%, a 46.50‑point gain over direct answering and 8.27 points over PhysMind.
ContPhy (multiple materials, continuous media): overall accuracy 70.56%, a 28.10‑point gain over direct answering.
Real‑World Evaluation
On 60 real‑world videos (billiards, bouncing balls, toy cars) with 60 questions, ATW achieves 71.67% accuracy versus 33.33% for direct answering, demonstrating potential for broader real‑scene deployment.
References
Project page: https://asking-the-world.github.io/
Code: https://github.com/zengsx23/Asking-the-World
Paper: https://arxiv.org/abs/2609.39135
PhysMind project page: https://physmind.github.io/
PhysMind code: https://github.com/ccyydd/PhysMind
PhysMind paper: https://arxiv.org/abs/2608.04575
Signed-in readers can open the original source through BestHub's protected redirect.
This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactand we will review it promptly.
How this landed with the community
Was this worth your time?
0 Comments
Thoughtful readers leave field notes, pushback, and hard-won operational detail here.
