Oneira: Interactive Video World Model with Persistent Object State Tracking

Introduction

Recent video world models can generate navigable scenes but lack a separate record of world state. Interactions often affect wrong objects, and changes disappear when the camera moves away. Existing simulators track state precisely but require pre-built assets, limiting open-ended generation.

Oneira Architecture

Oneira decomposes the problem into four components:

Coding Agent : Maintains a World State Table recording each object's semantic label, 3D bounding box, and interaction state. It uses off-the-shelf 3D perception models to initialize the table from the input image and updates it after each interaction.

World State Table : Stores only interaction-relevant information (walkable ground, object labels, 3D BBoxes, states). Object appearance is left to the video generator.

Engine : Renders the World State Table into a simplified conditional video where colored boxes represent objects; color changes indicate state transitions, disappearance indicates object removal.

Video Generation Model : Takes the first frame, conditional video, and per-chunk text descriptions to produce realistic video. It also uses memory frames when revisiting locations.

Interaction Workflow

Given a goal (e.g., "pick up the bottom-left apple"), the Coding Agent reads the World State Table, identifies the target instance and the chunk (each ~10-second segment split into 8 chunks) for execution. It infers secondary effects (other apples roll away) and includes them in the plan. The Engine updates the table accordingly, marking affected objects as moved. The video generator fills in motion details based on priors. After generation, the Coding Agent re-runs 3D perception on the new frame to register new objects (e.g., a newly appeared pear) and update positions of settled objects.

Key Design Choices

State and appearance are decoupled: the table tracks what changed , the generator handles how it looks .

3D BBoxes are category-agnostic (color only), enabling interaction with unseen object categories.

Memory frames preserve visual consistency when the camera returns to previous viewpoints.

Experiments

The authors introduce InteractionBench with 100 interaction test cases and 50 memory test cases. They compare against MiniMax-H3 Ref2VA, LingBot-World-V2, YUME 1.5, and AlayaWorld.

Instance-Level Interaction

Oneira achieves the best scores on three metrics: interaction occurrence (+29 pp), timing accuracy (+19 pp), and object correctness (+29 pp) over the strongest baseline. In scenes with multiple identical objects, baselines often manipulate the wrong instance or several at once; Oneira modifies only the specified one.

Long-Term State Retention

When the camera leaves and returns, Oneira improves PSNR by 93%, SSIM by 83%, and state retention rate by 18% compared to baselines. Video quality on VBench remains at or near the top.

Qualitative Capabilities

Demonstrations show open-ended interaction with newly generated content, multi-step manipulation sequences, and diverse interaction types (picking, pushing, opening).

Conclusion

Oneira validates that maintaining a separate, extensible World State Table enables precise, persistent interaction in generated video worlds. The approach outperforms existing methods on both interaction accuracy and long-term consistency.

Resources

Project Page: https://madaoer.github.io/projects/oneira/

Paper: https://arxiv.org/abs/2610.01614

Code: https://github.com/Madaoer/Oneira