ATAS: Agentic Video Generation with Shared World State Across Shots

Generating a coherent multi-shot video requires each shot to know where and when it occurs and what has already happened. Existing approaches that only pass prompts or recent frames struggle to provide a common reference for scenes and events separated by multiple shots.

ATAS Framework

ATAS (Any Time, Any Space) addresses this by letting video agents generate shots around a single, continuously updated world state. The framework decomposes a script into a set of scoped shot requests. Each request specifies not only what to shoot but also the scene, spatial location, camera, time interval, and, for re-shooting an earlier event, the corresponding event identifier. This allows a new shot to locate its place in the story.

Scene Memory for Spatial Reference

Accepted static geometry is registered into a common scene coordinate system, with dynamic regions handled separately. When a shot returns to the same room or views it from another position, the system can render a reference frame from the existing point cloud, giving the room layout and camera pose a reliable basis.

Event Memory for Temporal Reference

Character actions and object changes are not simply merged into the static scene. ATAS stores these dynamic observations indexed by scene, event, and time interval. To revisit an action from a new angle, the system retrieves the dynamic evidence from the corresponding moment rather than relying only on the most recently generated frames.

Memory Check After Generation

Each new clip is reconstructed and aligned to the existing scene. Only observations that pass review, registration, and state checks are written to persistent memory. A clip may be kept in the final cut, but if its geometry does not align, it will not serve as a reliable reference for later shots, preventing error accumulation.

The world state is not bound to a single agent. Agents responsible for different shots can query the same scene and event records on demand, reusing established spatial and temporal references instead of guessing the world from scratch.

Demonstrations

Two narrative examples illustrate the continuity:

Saturday Laundry in the Flat : A character puts down a cup in the living room, goes to the laundry room to start the washer, then enters the bedroom. When the camera returns to the living room, the sofa, coffee table, TV stand, and cup remain in their earlier positions. Back in the laundry room, the shot continues the washing process instead of generating an unrelated room.

Winter Fire in the Cabin : The character lights a fireplace, draws a heart on a frosted window, passes through a bedroom, and returns to the living room. The system must continue not only the room structure but also the results of the earlier actions — fire lit, heart drawn.

A multi-camera bathroom demo in the promotional video shows video frames, camera poses, and dynamic point clouds together: different views connect through cameras to a shared space, and actions have corresponding event timestamps. This demonstrates not merely extending a video but letting another shot revisit an existing event from a new viewpoint.

The project page provides an interactive timeline to inspect video, camera, and world state, and to observe which prior observations a shot depends on.

Experiments

ATAS was validated on four video generators: MiniMax-H3 , Wan3.0 , HappyHorse1.0 , and Wan2.7 . Compared to the agentic baselines used in the paper, ATAS improved the reported cross-shot consistency metrics in all four settings and achieved the highest overall benchmark score for each generator.

Separate tests evaluated spatial revisit (returning to the original space) and event re-shoot (re-filming a historical event from a new angle). ATAS outperformed compared methods on all three target-image spatial revisit metrics. In event re-shoot, it led on three of five criteria; character and object consistency tied for best, while specific action state reproduction was slightly below ReCA.

A user study across six dimensions — visual appeal, script fidelity, character consistency, background consistency, physical plausibility, and narrative coherence — showed ATAS achieving higher average ratings than the baselines.

The authors note that the shared spatial and event records provide a more stable basis for cross-shot generation, though the system is not a full physics simulator and results still depend on generator and reconstruction quality. Crucially, the question "where does the next shot come from, and which moment does it connect to?" now has a queryable record.

Code is open source, and the project page offers full videos and an interactive demo.