Code World Model: Letting Code Govern World Evolution While Video Models Handle Visuals

Westlake University's AGI Lab introduces Code World Model , a novel architecture that separates world evolution from visual rendering. The paper, titled "Code World Model: Coding Agent as World Brain" (arXiv:2608.25927), is authored by Yiwen Chen, Guosheng Lin, and Chi Zhang. Project page: https://buaacyw.github.io/cwm/, GitHub: https://github.com/buaacyw/code-world-model.

Why Video-Only World Models Fall Short

Current video world models predict the next visual frame from history but lack persistent rules, off-screen causality, and long-term memory. They treat the world as a sequence of observations, discarding the executable code that actually drives game logic (collision, faction relations, quest states). Video context windows are typically under one minute, while world events may evolve over days or years. Scaling video training adds more observed outcomes but does not recover the white-box mechanisms that generate them.

Core Idea: Code Runs the World, Video Shows It

The framework splits responsibilities:

Coding Agent ("world brain") reads the current world state, interprets interactions, decides which entities and mechanisms change, and composes, calls, or rewrites code.

Executable code acts as the Agent's low-latency extension, continuously updating positions, values, schedules, cooldowns, collisions, and rules without invoking the large model at every step.

Video model receives the resulting state via a conditioned interface and generates high-fidelity visual observations.

The Agent–Code loop is closed: execution results, tests, and new world feedback return to the Agent. World state is divided into executable state (programs, entity attributes, rules, relations, event history) and visual state (appearance, motion, temporal consistency). Both are coupled but updated by different mechanisms.

Proxy: A Programmable Visual Channel

To pass evolving state to the video model, the team introduces Proxy — a deterministic, coarse-grained visual condition compiled from world state. Proxy retains only the minimal constraints each frame must obey: entity positions, approximate scale, pose, motion trajectories, spatial relations, occlusion, and camera motion. It uses simple primitives (humanoid shapes, wireframe vehicles, low-poly vegetation, bounding boxes). Structured text handles identity, appearance, action semantics, and style; Proxy handles per-frame spatiotemporal constraints. Both trace back to the Agent-maintained state, keeping the pipeline inspectable and locally editable.

Proxy resolution is set to one-quarter of target video resolution (336×192 vs. 1344×768), yielding ~1/16 the visual tokens, adding limited inference overhead. The key is balancing constructability and constraint strength.

Data: Aligning Game Runtime and Real Video to the Same Interface

Training requires aligned Proxy and RGB video pairs. Games naturally provide this: the team recorded GTA V gameplay while simultaneously logging camera, entity, scene, and interaction states needed to compile Proxy. From a single run, multiple Proxy variants (different coverage, granularity) can be compiled without re-recording RGB. The same interface accommodates real-world data; a proof-of-concept on KITTI-360 used calibrated camera poses, semantic 3D reconstruction, and object annotations to offline-compile Proxy, while the model still receives only RGB target, Proxy video, and structured text.

Prototype Validation with ~5.6 Hours of Gameplay Video

Base video model : MiniMax-H3 Ref2VA

Training data : 157 gameplay clips, ~5.6 hours total, sampled at 2-second intervals into 9,420 five-second segments.

RGB target : 124 frames, 1344×768, 24 FPS.

Proxy : 336×192, combined with fixed log-depth and semantic ID maps.

Adaptation : Rank-128 LoRA across 50 Transformer blocks (~596M trainable parameters).

Hardware : 8× NVIDIA H800, 3 epochs.

Coding Agent : GPT-5.6 Sol, which composes, extends, and rewrites templates based on existing game engine code, basic player control, collision, and update loops.

Appearance anchor : GPT Image 2 generates the first-frame visual anchor from the initial Proxy and Agent-written text prompt; the full Proxy sequence then controls camera, entity motion, and scene layout.

Qualitative results demonstrate that after small-scale LoRA adaptation, the video model follows Proxy-specified positions, trajectories, layouts, and camera motion while preserving rich appearance and local dynamics. The project page includes full temporal comparisons against action-conditioned and camera-conditioned video world models, focusing on control granularity (latency not compared).

Long-Horizon Generation: Visual Style Can Change, World State Persists

For longer sequences, the system uses overlapping 124-frame windows (34-frame overlap, 90-frame stride). The previous window's tail RGB frames provide local continuity; a shared first-frame appearance anchor maintains global identity and look. Proxy conditions span the entire timeline, so even when visual style shifts periodically, entity trajectories, camera motion, and key spatial relations retain a continuous state source.

What This Work Fundamentally Changes

Code World Model shifts the responsibility division inside world models. Traditional video world models compress state, rules, memory, and visual prediction into a single generation stream. Here, executable world state moves to the center, managed by Coding Agent and code for causal evolution; only the portion that needs to be "seen" is selectively compiled into conditions. This decouples high-level reasoning (Agent, not at video frame rate) from low-level execution (code, no open-ended commonsense reasoning) and visual generation (video model, no need to learn full rules from scratch). Even if future multimodal networks unify language and video, an efficient, controllable interface for externally maintained persistent state — the "state-to-visual-condition" problem Proxy addresses — remains necessary.

Broader Implications

Beyond game generation, a world model that preserves rules, remembers off-screen changes, and produces high-fidelity observations can serve as environments for training and evaluating agents in embodied AI, autonomous driving, and long-horizon planning tasks that require understanding, predicting, and acting in dynamic environments.

Conclusion

If video generation models excel at imagining "what the world looks like," Coding Agent and code are better suited to answer "why the world became this way and how it should continue to change." Code World Model places both capabilities in a closed loop: knowledge and rules drive state; state enters visual space via Proxy; video model realizes the result as rich, open observations. The work is early-stage but argues that the path to open-ended world models may not lie solely in scaling video prediction, but in bringing executable code back to the core of world evolution — treating the world not as a stream of generated frames, but as a runnable system that can be understood, modified, executed, and leave lasting consequences.