LiveWorld: A New Paradigm for Video World Models that Keeps Off‑Screen Worlds Evolving

Background and Problem

Video world models are a key direction toward general intelligence because they can simulate a virtual environment that agents can explore, using visual priors from generative video models. However, existing approaches couple world evolution with the current camera view, so when an object leaves the field of view its state is frozen, violating the expectation of a continuously running world.

Out‑of‑Sight Dynamics and the Static‑World Assumption

The authors define the missing temporal process as Out‑of‑Sight Dynamics and point out that current models implicitly assume a static world —only content that enters the camera view continues to change. To break this assumption they propose LiveWorld , which explicitly decouples world evolution from observation rendering, allowing events to progress after they leave the view.

LiveWorld Design

LiveWorld maintains a global state independent of the camera. At each timestep the world state is first updated by an evolution operator , then a rendering operator combines the updated state with the camera pose and text conditions to generate the observed frame. Unlike prior methods that predict the next frame directly from past frames, LiveWorld separates the two processes.

The world is represented as a structured approximation: a largely static background stored as a global 3D spatial memory, and a small set of dynamic entities that retain a temporal dimension. Dynamic entities are updated in 4D point clouds, while the static background is incrementally fused into a global 3D point cloud using a feed‑forward SLAM framework (Stream3R).

Method Pipeline

Virtual Monitor Registration : Before each generation round, Qwen3‑VL and SAM3 analyze the previous video segment to detect potentially active entities (people, animals, vehicles). If a new entity appears in an uncovered region, a fixed‑position virtual monitor is created, recording the camera pose and frame as an anchor. The number of active monitors is capped; excess monitors are removed based on distance from the current observer.

Local Event Progression : Each monitor continues to generate video of its region after the camera looks away, using the anchored background, the cropped entity appearance, and a textual description of the next action. For example, a dog can finish eating and walk away instead of remaining frozen. Generated foreground video is re‑projected into 3D space, forming a time‑varying 4D point cloud.

Static Spatial Memory Accumulation : In parallel, background regions are extracted from observations and fused into a global 3D point cloud via Stream3R, providing a long‑term spatial foundation for revisiting scenes and changing viewpoints.

Rendering from Updated State : When the camera moves or revisits an area, the static 3D memory and the evolved dynamic 4D point clouds are projected onto the target camera trajectory, producing pixel‑level geometric conditions. A state adapter injects these conditions into a video diffusion model (Wan2.1‑14B‑T2V), while an Appearance LoRA supplies texture and identity details from retrieved reference frames, yielding videos that follow the camera motion and reflect off‑screen evolution.

Experiments and Benchmark

The authors construct LiveBench , the first benchmark targeting out‑of‑sight dynamics, containing 100 scenes and 400 evaluation sequences with multi‑round camera trajectories and textual event scripts. Two revisit trajectory types are used: Same‑Pose (A→B→A→B→A) for long‑term state change, and Different‑Pose (A→B→C) for cross‑view consistency.

LiveWorld is compared against Matrix‑Game‑2.0, Hunyuan‑GameCraft‑1.0, and Spatia. On Same‑Pose long‑term revisits, LiveWorld achieves a VQA‑Acc of 54.620 , far surpassing Spatia (14.655), GameCraft‑1 (10.273) and Matrix‑Game‑2.0 (5.012). On the more challenging Different‑Pose revisits, LiveWorld still reaches 49.478 while other methods drop to single‑digit scores.

For identity and geometry consistency, LiveWorld attains a foreground DINO similarity of 0.721 (vs. 0.416 for Spatia) and a dynamic point‑cloud Chamfer Distance of 0.135 , outperforming all baselines. Background consistency is comparable or better than Spatia, which uses explicit 3D memory.

In multi‑event scenarios, removing the event‑evolution module reduces Full Success to 0 %, whereas the full LiveWorld reaches 26 %. Ablation studies show that removing spatial memory harms camera control and causes drift, while removing reference frames leads to gradual loss of foreground identity and background appearance, confirming that improvements stem from the combined system design rather than merely a larger generative model.

Conclusion and Outlook

LiveWorld demonstrates that decoupling world evolution from observation rendering enables video world models to maintain and advance the state of off‑screen objects, moving beyond “remembering past frames” toward truly persistent world simulation. The work provides a practical, evaluable baseline for continuous world modeling, while future research should explore implicit dynamic memory, more efficient 4D representations, better state‑injection mechanisms, and scalable cross‑region event interaction.