MagicWorld: A Framework for Stable Long‑Horizon Interactive Video World Modeling

Research Background

Video World Models aim to generate continuous visual dynamics conditioned on user actions for interactive exploration, scene prediction, and long‑term planning. Existing models exhibit two critical failure modes during prolonged interaction:

Motion drift : dynamic entities such as pedestrians or vehicles become static or move unrealistically.

Long‑horizon instability : autoregressive errors accumulate, leading to structural distortion, semantic drift, and overall scene collapse.

MagicWorld Framework

MagicWorld addresses these failure modes with three tightly coupled components:

Flow‑guided motion preservation

History cache retrieval

Multi‑shot aggregated DMD training with dual‑reward weighting

1. Flow‑guided Motion Preservation

The model first predicts denoised latent representations via flow‑matching. Adjacent latent frames are warped using optical flow computed in latent space, and higher weights are assigned to regions with large motion magnitude. This imposes stronger temporal‑consistency constraints on truly moving areas while leaving static background largely unconstrained, thereby reducing motion drift and producing smoother trajectories.

2. History Cache Retrieval

At each autoregressive step the generated latent feature is written to a history cache. In the subsequent step the current latent is compared against all cached latents using a latent‑space similarity metric; the top‑scoring historical states are selected as auxiliary conditions and injected into the generation process. Because retrieval operates on semantic similarity rather than temporal proximity or explicit camera geometry, the model can recall the most relevant past scene regardless of how many steps have elapsed, preserving structural consistency and mitigating long‑term drift.

3. Multi‑shot Aggregated DMD Training

Instead of updating parameters after every interaction step, MagicWorld simulates a multi‑step rollout, aggregates the distillation losses across the entire sequence, and performs a single optimization pass. This encourages the model to optimize the whole interaction trajectory rather than isolated steps.

4. Dual‑Reward Weighting

Training incorporates two reward signals—visual quality and motion quality—and combines them with a weighting scheme that balances the two objectives. The combined loss drives the model to generate both clear frames and temporally coherent motion over long horizons.

RealWM120K Dataset

To evaluate long‑term interactive video modeling, the authors constructed RealWM120K, a city‑walk dataset covering multiple global cities, seasons, times of day, and weather conditions. Each clip is annotated with text descriptions, camera trajectories, point clouds, object masks, and depth maps, providing a realistic benchmark for dynamic scenes and non‑trivial camera motion.

Performance Evaluation

On the RealWM120K validation split, MagicWorld achieves an Overall Score of 0.8547 (the highest among compared methods) and an inference latency of 15 seconds , demonstrating competitive efficiency. Qualitative comparisons show markedly improved motion realism and temporal consistency across diverse scenes.

Paper: https://arxiv.org/abs/2511.18886v2