Reading, Writing, Sharing, Interacting: A Unified Survey of Video World Models

Paper Information

Title: Reading, Writing, Sharing, and Interacting with the World through Video

Authors: Shengqiong Wu, Meng Luo, Bobo Li, Ziwei Liu, Shuicheng Yan, Ming-Hsuan Yang, Philip Torr, Hao Fei (first two co-first authors; Hao Fei project lead and corresponding author)

Version: arXiv v1, submitted 22 October 2026; 102 pages, 17 figures, 19 tables. No conference acceptance noted.

Links: arXiv abstract https://arxiv.org/abs/2610.31668, PDF https://arxiv.org/pdf/2610.31668, project page https://vid-gen.github.io/video-world-model-survey, resource repository https://github.com/vid-gen/awesome-video-world-model

1 Introduction

Video captures entities, relations, and actions evolving over time, often combined with audio, language, and tool feedback. Research has progressed from action recognition and spatiotemporal encoders to video-language models, large-scale generators, unified understanding-generation systems, and action-conditioned prediction with interactive simulation. These share a focus on the observation–world relationship, yet using similar video data does not imply equivalent capabilities. A generator can synthesize clips from descriptions; a world model must also predict how states evolve under conditions, actions, and feedback. What the model design supports, what the training data supervises, and what evaluation truly proves must be kept separate.

2 The World-Modeling Lens

2.1 World, Video, and Model State Representation

The latent world contains entities, attributes, relations, and dynamical variables. Video is produced by a camera and perception process, showing only a partial, lossy, temporally ordered observation. Different true states can yield the same frame; compression, editing, and resolution lose further information. A model's state built from history and actions is not the true world state—it may be explicit geometry, video tokens, or hidden features, and may be incomplete. Hence "having a latent vector" is not evidence of recovering physical state.

2.2 Two Kinds of Consistency

Temporal coherence concerns motion, appearance, and flicker between adjacent frames. World consistency concerns whether an entire trajectory respects identity persistence, spatial relations, dynamics, and specified interventions. The two do not imply each other: an object can leave the frame, change identity, and return smoothly; a film's hard cuts are not smooth but can depict a consistent world. World consistency is a formal evaluation target, not a directly computable universal metric; it requires concrete tests like occlusion, revisiting, and intervention.

2.3 Four Operations

Read: estimate semantic, spatial, temporal, physical, or causal attributes from video.

Write: generate video from partial conditions (text, image, audio, trajectory, layout).

Share: connect parameters, representations, interfaces, or objectives between understanding and generation.

Interact: update predicted state with actions and new observations.

These are not mutually exclusive labels. An interactive system also reads history and generates frames. Sharing must distinguish "one system does two tasks," "genuinely shared components," and "functional synergy under matched resources"—they should not be conflated.

3 Preliminaries

3.1 Video Scenarios

Short clips, long videos, egocentric, multi-camera, multimodal, and streaming inputs impose different state requirements. Offline models see the whole clip; online models only current and past observations; driving and robotics face action-frequency, latency, and safety constraints. Evaluation conclusions must state the observation conditions.

3.2 Video Large Language Models

Typical systems use a video encoder to extract frame or clip features, pass them through a connector into a language model, then align and instruction-tune to answer questions. Sampling, compression, temporal encoding, and memory determine which evidence actually enters reasoning for long videos. Correct language answers do not guarantee video grounding: text priors, single frames, and static appearance can form shortcuts, requiring dedicated controls.

3.3 Generative Modeling Foundations

Diffusion learns denoising; flow matching learns the velocity field from noise to data; autoregressive models predict discrete or continuous units in order; masked models recover masked positions from visible context. The generation process's step count is not the same variable as time in the video world.

3.4 Taxonomy of Video World Models

The paper uses two independent axes: prediction space (decodable observation vs. video-derived latent state) and transition interface (passive prediction, fixed-action conditional prediction, or interactive with new feedback and actions). This yields six categories, avoiding labeling all video generators as interactive world models.

4 Video Comprehension: Reading the World

4.1 Tasks, Data, and Evaluation

Understanding tasks are grouped by world attribute: semantic (objects, actions, events), spatial (location, identity, geometry, occlusion), temporal (order, duration, boundaries, long-range dependencies), physical (contact, stability, containment, dynamics), and causal (action effects, counterfactuals). Benchmarks like Kinetics, Video-MME, VSI-Bench, TemporalBench, Physion, and CLEVRER provide different evidence. A single aggregate QA score cannot replace all dimensions. Text-only, single-frame, frame-shuffled, and context-limited ablations reveal whether the model truly uses video and time. Near-duplicate frames with opposite answers further exclude appearance shortcuts.

4.2 Video Representation Learning and Encoding

Contrastive learning pulls matching video/text together—good for semantic alignment.

Masked learning reconstructs pixels or features.

Predictive learning forecasts future or missing representations from context.

Generative learning retains information through reconstruction, continuation, or captioning.

Categories can overlap; classification is by primary supervisory objective. Reconstruction ability, text alignment, and recoverable physical state are not equivalent. Dropping texture may help recognition but hurt generation; optimizing only language matching may lose quantitative physical variables.

4.3 Video-Language Foundation Modeling

Architecture must balance video encoding, visual token compression, and language reasoning. Training typically goes through vision-language alignment, instruction tuning, and reasoning reinforcement; at test time evidence search, segment-wise analysis, or reasoning budget can be added. More compute only helps when it supplies missing evidence; longer language reasoning chains do not directly imply more accurate world knowledge. Long-video models should report actually usable history, not just maximum input length.

4.4 Agentic Video Modeling

Video agents turn watching into an iterative workflow: propose sub-questions, locate clips, call detection, tracking, ASR, or retrieval tools, then aggregate evidence. External memory helps with ultra-long history and repeated queries. The cost is that tool outputs, retrieval, and intermediate summaries can propagate errors. Timestamps and sources should be preserved; evaluate evidence hit rate, call cost, and final judgment—not just answer fluency.

4.5 Applications

Embodied and egocentric perception must maintain object, goal, and action context; autonomous driving must disentangle ego-vehicle, camera, and other-agent motion; medical and industrial domains require professional constraints and miss-rate control. When transferring to decision tasks, check actual utility rather than reusing generic QA metrics.

5 Video Generation: Writing the World

5.1 Tasks, Conditions, and Evaluation

Text, image, video prefix, audio, camera, trajectory, and layout specify different conditions; the model completes the unspecified parts. Evaluation must separate perceptual quality, condition adherence, temporal coherence, and world consistency; physical constraints and control response need dedicated tests. Two equally pretty videos may differ in action adherence; single-clip quality does not prove cross-shot, occlusion, or revisit state retention.

5.2 Video Latent Representation Learning

Continuous latents reduce pixel-generation cost; discrete tokens suit sequence modeling; hybrid or semantically aligned representations try to serve both understanding and generation. Compression must trade off reconstruction detail, motion, length, and semantic organization. Selecting an encoder solely by reconstruction error may miss structure needed for downstream generation. Overly abstract semantic representations lose appearance; over-retaining pixels increases sequence cost.

5.3 Generative Video Foundation Modeling

Diffusion and flow models suit continuous latents; autoregressive and masked prediction offer different ordering and parallelism. They can combine with language-model planning, storyboarding, and agent-based iterative critique: generation mechanism and system workflow are two layers of choice, not mutually exclusive categories. Autoregressive models accumulate prefix errors; chunked generation must maintain cross-chunk identity and motion; multi-step sampling faces interaction latency. Comparisons should state resolution, length, sampling steps, and hardware.

5.4 Control Strategies

The paper categorizes by intervention point: attention control, adapters, latent space, concatenation/context conditioning, guidance, and fine-tuning. Camera, pose, object identity, and motion signals can enter via multiple routes; a single system may combine them. Effective control should change the target attribute, preserve unrelated attributes, and sustain the effect. Altering one motion while letting background or identity drift is not success even if visual quality is high. Combined controls must be checked for mutual interference.

5.5 Applications

Applications include creative production, character animation, embodied video planning, and driving/robot synthetic data. A fixed generated plan cannot automatically respond to new feedback, unlike an online-updatable simulator. Synthetic data should be validated by real perception or policy training gains; a clip with visual motion does not equal correct robot action labels. Generators and inverse-dynamics annotators may jointly amplify errors.

6 Unified Video Models: Sharing the World

6.1 Unified Architectures

Discrete token approaches share sequence and autoregressive backbone, but quantization and long sequences become bottlenecks.

Continuous approaches share latent space, usually keeping separate output heads.

Modular approaches use a semantic bridge to connect understanding and generation experts.

Agentic approaches share task state and orchestrate perception, generation, editing, and review tools.

Emu3, Show-o2, NExT-GPT, UniVideo represent different integration points. Modular designs ease component swapping but may lose geometry and motion in the bridge; shared backbones do not automatically resolve the tension between understanding's invariance needs and generation's detail needs.

6.2 Learning Paradigms

Independent learning connects separately trained experts—useful as ensemble baselines.

Dual learning uses bidirectional reconstruction or cycles.

Joint learning optimizes shared components together.

Collaborative learning requires one direction to demonstrably help the other.

Cycles are not exact inverse mappings: one description permits many videos; constraints should preserve task-relevant identity, events, viewpoint, and motion while tolerating plausible appearance variation. Joint training improving both scores may just use more data and compute; matched-resource and switchable ablations are needed to confirm cross-directional information contribution.

6.3 Applications

Instruction editing understands then generates modifications; generative reasoning feeds predicted intermediate states to subsequent analysis; perception-generation-planning systems use imagined results for action selection. Intermediate products must be inspectable—answers cannot be hidden in generation conditions and then claimed as reasoning gains. Tool chains must test how erroneous captions, retrieval, or critiques affect final output.

6.4 Evaluation

First evaluate understanding and generation separately, then interleaved, multi-turn, and collaborative tasks. Capability coverage only proves "both can be done"; shared parameters only prove "structure connected"; collaboration must prove "connection brings gain." Comparisons should fix task, data, model capacity, compute, and observation budget; ablate by closing target-passing paths; check whether temporal order, action, identity, and state improve. Image-domain collaboration conclusions cannot directly substitute video temporal evidence.

7 Video World Models: Interacting with the World

7.1 Architectures and Training Paradigms

Observation space can predict future frames or video tokens directly; latent space predicts features and plans as needed. Both combine with passive, action-conditional, and online interactive modes. Passive models learn dynamic priors but do not identify specified action effects; action-conditional models receive a fixed control sequence before prediction; interactive models receive new actions and feedback during rollout. Latent prediction need not decode pretty video—if it retains decision-relevant state it can be valuable. Conversely, generating action-conditioned video does not imply closed-loop control capability.

7.2 Core Technical Challenges

Action control must disentangle agent action, camera motion, and background motion.

Long-term memory must maintain occluded identity and off-screen changes, not just replay old frames.

Real-time response must meet control budget, not just offline high frame rate.

Physical causality requires that changing a condition alters the target trajectory while camera, background, and unrelated attributes stay constant.

Retrieval memory, recurrent state, rollout training, causal generation, and few-step sampling address different bottlenecks. Increasing history capacity may not fix off-screen dynamics; distillation speedup does not prove dynamics accuracy; benefits and degradations must be checked separately.

7.3 Applications

World models can serve as synthetic data engines, imagination training environments, policy evaluation environments, predict-then-act planners, or human-facing interactive simulators. Application evidence cannot stop at video quality. Training environments should improve real policy performance; evaluation environments should correctly rank candidate policies; planners should improve action decisions. Frequent replanning may compensate for predictor defects, so fixed-candidate action ranking diagnostics should also be reported.

7.4 Benchmarks and Evaluation

Challenge-oriented tests check control, memory, latency, and physics; application-oriented tests check training, planning, policy evaluation; comprehensive suites cover more world attributes but different prompts, prediction horizons, action spaces, and judges make direct cross-comparison unreliable. Observation models need frame and trajectory tests; latent models need state probes; both ultimately need decision utility. Reports should include rollout length, action frequency, hardware, resolution, sampling steps, latency variance, and control for data leakage, judge bias, and environment overlap.

8 Open Challenges and Future Directions

8.1 Long-Horizon World Consistency

Reading distant evidence, remembering invisible states, and letting them evolve correctly are distinct capabilities. Ice melts during occlusion; replaying old observations restores outdated states. Future memory should predict hidden changes and combine with training that suppresses rollout error accumulation. Evaluation should vary rollout length and revisit distance separately, report degradation curves, memory size, and latency; distinguish rendering drift from state loss—not just a final aggregate score.

8.2 Physical and Causal Grounding

Recognizing physical laws, generating plausible phenomena, and correctly responding to physical interventions require progressive evidence. Fix initial state, change only gravity or one action; the target trajectory should change while camera, background, and unrelated attributes stay constant. Physical priors, simulation, geometric rewards, and retrieval can provide supervision but are limited by external information quality. Use paired interventions to locate the first violated constraint and time, rather than a generic physical plausibility score.

8.3 Measurable Synergy in Unified Video Models

The open question is whether gains come from information sharing or extra data, parameters, and supervision. Cross-directional paths should be designed as independently closable components; compare with matched resources whether event understanding improves motion generation, whether generating intermediate states improves temporal reasoning and planning.

8.4 Standardized Evaluation of World Properties

Evaluation protocols must match prediction space and interaction interface: passive prediction tests unseen transitions; action-conditional tests fixed control response; interactive models test mid-rollout intervention and control-to-display latency. Video and feature models use different probes but should face common decision tests. Standardize data loading, planners, and environment variations; use human annotations to verify learned judges. Reporting only FPS cannot explain whether control response is timely or stable.

8.5 Scaling Control from Unlabeled Video

Internet video lacks true action labels. Latent actions learned for reconstruction may simultaneously encode gripper push, camera pan, and background motion; thus reconstruction accuracy and codebook utilization are not sufficient evidence for control semantics. Few-shot annotations can anchor true control; multi-view and intervention pairs can disentangle irrelevant variations. Pseudo-actions are only valid if they produce expected effects in controlled environments or on real robots; tests must cover unseen cameras, backgrounds, scenes, and bodies.

8.6 Cross-Scene and Cross-Embodiment Generalization

A new room and a new robot are not the same generalization. Body, sensors, action frequency, and degrees of freedom change the physical meaning of the same command; a familiar robot in a new scene cannot be written as unknown-embodiment zero-shot transfer. Future work should align action effects by object displacement, contact, and task outcome; cross-hold-out scene and body combinations; specify zero-shot, few-shot calibration, or full fine-tuning; report target data volume and safety performance.

8.7 Stochastic Worlds and Rare Events

The same history and action can have multiple valid futures. Generating a collision on prompt gives one case, not the true collision probability. Diversity does not replace calibration; multiple samples taking the best hides error probability mass. Future work should constrain outcome distributions with real frequencies or plausible simulation priors; report calibration, mode coverage, and proper scores under fixed sampling budget; distinguish invalid from uninterpretable samples. Rare cases must be checked for physical feasibility, causal relevance, and real training utility.

9 Conclusion

The survey treats video as observation trajectories of a dynamic world, using read, write, share, and interact to explain how models connect observations and states. It does not upgrade "looks real" to "understands the world"; instead it demands stronger evidence for stronger claims. Future progress depends not only on more video but on occlusion-revisit evidence, paired physical changes, shared-path ablations, true action labels, cross-scene embodiment holdouts, and event-frequency interventions. Long-term state, controllable effects, real-time closed-loop, and decision gain should become goals alongside visual fidelity.

Key takeaway: Temporal coherence belongs to the video trajectory; world consistency belongs to the state behind the trajectory; a true world model must turn predictions into better actions.

Code example

来源：专知
本文
约7000字
，建议阅读
10
分钟
时间连贯属于视频轨迹，世界一致性属于轨迹背后的状态；真正的世界模型还要让预测转化为更好的行动。