Code World Model: Coding Agent Governs World Evolution, Video Model Renders Visuals

The article introduces Code World Model (CWM), a research framework from Westlake University's AGI Lab that rethinks video world models by dividing labor between a Coding Agent and a video generator. Current video world models only predict the next visual frame from history, lacking explicit mechanisms for rules, memory, off-screen causality, and long-horizon evolution. CWM argues that an open world needs two distinct capabilities: low-frequency complex reasoning (understanding events, planning consequences, modifying mechanisms) and high-frequency repetitive execution (updating positions, values, cooldowns, collisions). Putting both into a single video model is inefficient and hard to make reproducible.

Core Architecture: Coding Agent as World Brain

The Coding Agent reads the current world state, interprets new interactions or events, decides which entities and mechanisms must change, and chooses to call existing code or locally rewrite the world program. Code acts as the Agent's executable extension, continuously performing dense, deterministic, reusable state updates without invoking a large model at every step. The Agent can compose, invoke, or modify code based on new situations, changing not only the current state but also how the world will run in the future. Execution results, tests, and new world feedback return to the Agent, forming a continuous Agent–Code loop.

The world state is split into two parts: an executable state that stores programs, entity attributes, rules, relationships, and event history; and a visual state that stores appearance and motion information generated by the video model and must remain temporally consistent. The two are coupled: code ensures rules and consequences persist, while the video model leverages appearance, motion, and interaction priors from large-scale visual data to realize the world state as high-quality observations.

Proxy: A Programmable Visual Channel

To pass the evolving state to the video model, the authors propose Proxy — a coarse-grained visual condition deterministically compiled from the world state. Proxy does not aim to produce a low-quality final frame; it only retains the minimal constraints the current observation must obey: entity positions, approximate scale, pose, motion trajectories, spatial relationships, occlusion, and camera motion. Simple humanoids, wireframe vehicles, low-poly vegetation, or bounding boxes serve as callable base components. Structured text handles identity, appearance, action semantics, and style; Proxy handles per-frame spatial and temporal constraints. All control signals trace back to the Coding Agent and the code-maintained world state, keeping the state–condition pathway inspectable, addressable, and locally modifiable.

Proxy granularity balances constructability and constraint strength. Overly rich Proxies burden the Agent with excessive joint and detail trajectories; overly sparse ones may insufficiently constrain the video model. The current implementation sets Proxy resolution to one-quarter of the target video in both dimensions, so visual token count is roughly one-sixteenth of the target, adding limited inference overhead.

Data Pipeline: Aligning Game Runtime and Real Video

Training requires strictly aligned Proxy and RGB videos. Games are a natural fit: the team recorded GTA V gameplay while simultaneously logging camera, entity, scene, and interaction states needed to compile Proxies. Code can recompile Proxies with different coverage and granularity from the same run without re-capturing RGB. The same interface accommodates real video; a proof-of-concept on KITTI-360 used calibrated camera poses, semantic 3D reconstruction, and object annotations only for offline Proxy compilation. The video model ultimately receives RGB target, Proxy video, and structured text; camera motion and action consequences are already embedded in Proxy, eliminating separate action labels.

Prototype Validation

Data: 157 gameplay segments, ~5.6 hours total; sampled at 2-second intervals yielding 9,420 five-second clips.

Video specs: RGB target 124 frames, 1344×768, 24 FPS; Proxy 336×192 with fixed log-depth and semantic ID maps.

Model: MiniMax-H3 Ref2VA adapted via rank-128 LoRA across 50 Transformer blocks (~596M trainable parameters).

Training: 3 epochs on 8 NVIDIA H800 GPUs.

Inference: GPT-5.6 Sol as Coding Agent, building on existing game engine code, basic player control, collision, and update loops. GPT Image 2 generates appearance anchors from the first-frame Proxy and Agent-written text prompts; the full Proxy sequence continues to control camera, entity motion, and scene layout.

Qualitative results show the adapted video model follows Proxy-specified character positions, action trajectories, scene layouts, and camera motion while preserving rich appearance and local dynamics. The project page provides full temporal comparisons against action-conditioned and camera-conditioned video world models to highlight Proxy's per-frame control granularity (latency not compared).

Long-Horizon Generation

For longer sequences, the system uses overlapping 124-frame windows: adjacent windows overlap 34 frames, each step advances 90 frames. The previous window's tail RGB frames provide local continuity; the same first-frame appearance anchor maintains global identity and appearance. Proxy conditions span the full timeline, so even when visual style cycles, entity trajectories, camera motion, and key spatial relations retain a continuous state source.

Implications and Outlook

CWM shifts the responsibility division inside world models: executable world state moves to the center, managed by Coding Agent and code for causal evolution; only the portion that needs to be "seen" is selectively compiled into conditions. This decouples high-level reasoning from low-level execution: the Agent need not operate at video frame rate, code need not possess open-ended commonsense reasoning, and the video model need not learn complete rules from scratch — it only implements explicit world states as visual observations. Even if future multimodal networks unify language and video, an efficient, controllable input interface for externally maintained persistent world state remains necessary; the "state-to-visual-condition" problem Proxy addresses persists.

Applications extend beyond game generation: an open world that preserves rules, remembers off-screen changes, and generates high-fidelity observations can serve as environments for training and evaluating agents in embodied AI, autonomous driving, and long-term planning tasks requiring understanding, prediction, and action in dynamic environments.

"If video generation models excel at imagining 'what the world looks like', Coding Agents and code are better suited to answer 'why the world became this way, and how it should continue to change.' Code World Model attempts to place these two capabilities in the same loop: let knowledge and rules drive state, let state enter visual space via Proxy, and let the video model realize the result as rich, open observations."

The work is early-stage but posits a notable judgment: the path to open-ended world models may not lie solely in scaling video prediction models, but in bringing executable code back to the core of world evolution. The world becomes not just a continuously generated sequence of frames, but a runnable system that can be understood, modified, executed, and leave persistent consequences.

Paper: arXiv:2608.25927 | Project page: https://buaacyw.github.io/cwm/ | GitHub:

https://github.com/buaacyw/code-world-model