Code-as-World: Reconstructing the Physical World from Video

Current AI models can generate photorealistic video, but they only predict how the world looks; they do not explain why physical processes unfold as they do. The next step, therefore, is to move from pixel‑level prediction to uncovering the underlying mechanisms that generate those observations.

Code-as-World Framework

Code-as-World proposes representing a physical scene as executable code that explicitly encodes three interrelated aspects:

Composition : what exists in the world – objects, geometry, mass, friction, gravity, etc.

Evolution : how the world changes – states, motions, interactions, and key events over time.

Appearance : how the world is observed – camera parameters, background, materials, lighting.

The resulting "world hypothesis" can be queried, executed, intervened upon, and verified, turning a static scene description into a dynamic, testable model.

Why Existing Representations Fall Short

Pixels preserve rich detail but hide causal information; 3‑D reconstruction restores geometry but does not explain forces; natural language can express entities and relations but struggles with continuous trajectories and precise physical parameters. Code-as-World aims to combine the semantic richness of language, the structural fidelity of 3‑D, and the temporal continuity of video.

Agentic Discovery Loop

The system treats world reconstruction as an iterative discovery process consisting of five stages:

Propose : generate or modify a world hypothesis from current evidence and feedback.

Instantiate : compile the hypothesis into an executable program.

Execute : run the simulation to obtain a full state trajectory.

Render : convert the trajectory back into observable video.

Verify : compare rendered video with the original evidence, producing feedback for the next iteration.

This loop mirrors abductive reasoning in scientific discovery, continuously refining the hypothesis until it explains the observations while remaining parsimonious.

Scalable Physical Supervision

Because the executable world contains explicit physical parameters, it provides dense supervision that raw video lacks. The authors trained Code-as-World‑VL models and evaluated them on the QuantiPhy benchmark (proposed by Fei‑Fei Li’s group). The results are:

Code-as-World‑VL‑9B outperforms Gemini‑3.1 Flash by 54.8 points. Code-as-World‑VL‑27B Reasoning reaches 58.6 points.

These numbers demonstrate that a code‑based world representation can serve as a powerful source of physical supervision for vision‑language models.

Code vs. Natural Language

Code offers a precise, executable formalism that can encode variables, constraints, and rules for exact simulation. Natural language provides broader semantic coverage, capturing entities, causal intuition, and long‑term reasoning. Together they form complementary layers of abstraction: code for exact execution, language for wide‑range understanding.

Unified World Representation for Reasoning, Generation, and Interaction

With a single executable world model, the same representation supports:

Physical reasoning – query and infer within the structured world before answering visual questions.

Video generation – evolve the world state first, then render pixels, preserving object identity and geometry.

Embodied interaction – simulate multiple futures, select actions, and bridge simulated and real environments.

Thus, querying, evolving, rendering, and intervening become operations on a shared world model rather than isolated modules.

Conclusion

Code-as-World illustrates a deeper vision of Physical RSI: a manipulable, accumulable, and correctable representation of physical experience that can be encoded, reproduced, verified, and fed back into learning systems. While scaling language models has already exploded, scaling physical intelligence now hinges on expanding executable, structured physical knowledge.