Oxford Survey: One World, Four Operations for Video Foundation Models

Core Framework: One World, Four Operations

The survey treats a video V as a partial, lossy, temporally ordered observation of an underlying temporal world W. The model maintains an internal representation state S_t, which generally does not equal the true world state W_t. All work is categorized into four operations based on the model's relationship with the world, not input/output modalities:

Reading : Infer state from observations (video understanding).

Writing : Construct video from conditions (video generation).

Sharing : Couple understanding and generation via shared tokens, latent spaces, or interfaces.

Interacting : Predict world evolution under actions, supporting reasoning and planning (world models).

A single system can perform multiple operations simultaneously (e.g., a unified editor reads then writes; an interactive world model reads history, writes next frame, updates state in one backbone).

Two Consistencies Must Be Separated

Temporal coherence : Local consistency between adjacent frames (no flicker, no unreasonable jumps).

World consistency : The entire video corresponds to a valid world trajectory—identity, layout, dynamics, physics, and causal constraints all hold.

A video can be perfectly smooth yet world-inconsistent (e.g., a car behind a wall vanishes). Evaluating world models only with temporal coherence is a common pitfall.

Evidence Ladder: Stronger Claims Require Stronger Evidence

To avoid overclaiming that good performance on a single task equals learning a world model, the survey defines four evidence levels for each operation. Higher-level world-modeling claims demand higher-level evidence.

Panorama: Knowledge Graphs for Four Domains

Each operation is broken into technical streams with representative works:

Reading : Spatiotemporal encoders, representation pre-training, video-language foundation models, long-video reasoning.

Writing : Tokenizers & latent representations, diffusion/flow generation, autoregressive/masked generation, controllable generation & editing.

Sharing : Discrete token unification, continuous/hybrid latent unification, decoupled/modular unification, agent/orchestration unification.

Interacting : Observation-space rollout, fixed-plan prediction, latent-state planning, action controllability, long-term memory, real-time response, physics/causal grounding, data engines & evaluation.

Reading: Bottleneck Is State Grounding, Not Context Length

Video representation learning splits into four families: contrastive learning (video-video, video-text), masked learning (pixel/feature reconstruction), predictive learning (latent-space prediction of masked parts, e.g., V-JEPA), and generative learning (visual continuation, captioning). Video-LLMs map visual evidence to language; video agents decide where to look and what tools to call.

Key observation: Encoders compress observations, Video-LLMs map evidence to language, agents select what to watch—their shared bottleneck is not how many frames they can accept, but how much state they can maintain and ground. Longer context only helps when it corrects or stabilizes state estimation. Evaluation should ask: Does new evidence improve state estimation? Is it traceable to source segments? Can it withstand follow-up questioning?

Writing: Rendering Fidelity and Stateful Control Are Two Independent Axes

Generation starts with choosing a latent representation: continuous latent spaces (per-frame VAE, spatiotemporal VAE, causal VAE), discrete tokens (VQ, residual quantization, codebook-free like FSQ/LFQ/BSQ), hybrid representations, and semantics-aligned representations (serving both understanding and generation).

Four generation paradigms: diffusion, flow matching, autoregressive, masked prediction. The survey unifies notation (Z_K → Z_k → Z_{k-1} → Z_0) and discusses combinations for long video (Diffusion Forcing, Self Forcing).

Control mechanisms are classified by where intervention occurs: input-level conditioning (channel/token/context concatenation), attention manipulation, latent-space control (partial noising, inversion, latent condition construction), adapter/residual injection, parameter/embedding fine-tuning, sampling guidance (e.g., classifier-free guidance).

Key insight: Video priors make clips increasingly photorealistic, but visual fidelity and stateful control are two separate goals. Good control should change only its target, persist over time, and compose with other controls without breaking identity, geometry, or motion. Progress should be measured by comparing controlled vs. uncontrolled matched rollouts, not by single-rollout quality alone.

Sharing: Synergy Requires Controlled Cross-Directional Transfer

Unified models are grouped by what is shared:

Discrete token unification : Shared token sequence & autoregressive backbone (Emu3, Show-o).

Continuous representation unification : Shared continuous multimodal representation (Show-o2, TUNA).

Decoupled/modular unification : Shared semantic interfaces (NExT-GPT, Omni-Video).

Agent/orchestration unification : Shared task state & multimodal artifacts (Mora, VISTA).

Training modes: isolated learning, dual learning (cyclic supervision), joint learning (shared parameters), collaborative learning (explicit, verified cross-directional transfer).

Evaluation is split into three layers: capability coverage (both directions work), architectural sharing (what is actually shared), functional synergy (does one direction improve the other under matched resources).

Critical point: Shared tokens or parameters only prove coupling; separate scores only prove capability; mixed tasks only prove interoperability. Proving synergy requires controlled ablation: match data, capacity, compute, ablate cross-directional paths, measure changes in generation, understanding, or integrated tasks.

Interacting: 2×3 Taxonomy of Video World Models

Two axes classify video world models:

Prediction space : Predict observations (frames, video tokens, decodable latents) vs. predict latent states (video-derived features, no rendering required).

Transition interface : Passive (no action input), action-conditioned (control fixed before prediction), interactive (accepts new control after feedback).

The six cells correspond to: video prediction (SV2P), controllable video prediction (GAIA-1, Vista, IRASim), advancing visual worlds (Genie, DIAMOND, GameNGen), masked feature prediction (V-JEPA/V-JEPA 2), candidate feature rollout (DINO-WM, V-JEPA 2-AC), plan-act-replan (MPC closed-loop). A model can occupy multiple cells; attention pattern does not determine column—if it accepts new control between output frames, it is interactive.

The chapter further discusses control, memory, real-time response, physics/causal grounding, applications (data engines, policy training, evaluation), and evaluation protocols.

Evidence must match the transition protocol. Passive models show structural prediction; controlled models must change the world where the action applies and keep elsewhere unchanged; interactive systems must preserve those consequences under feedback; decision applications must turn them into better choices.

Seven Open Challenges

Long-horizon world consistency : Hidden states evolve during occlusion; revisiting old observations misses those changes (e.g., occluded ice melting).

Physics and causal grounding : Changing gravity should alter trajectories only, not unrelated scene properties.

Measurable synergy : Two resource-matched systems differing only in whether understanding and generation exchange information.

Standardized evaluation of world properties : Video prediction and feature prediction need different probes but must face the same decision tests.

Scaling control from unlabeled video : Action effects are entangled with camera and background motion.

Cross-scene and cross-embodiment generalization : Held-out scene-body combinations test whether familiar factors compose and action effects persist.

Stochastic worlds and rare events : Valid futures differ in frequency; invalid and unreadable samples must be counted separately.

Scaling world models needs intervention, not just more clips. Passive video expands coverage, but hidden states, physical causes, cross-directional transfer, action semantics, and outcome probabilities remain underdetermined. Observation scale tells us what co-occurs; intervention tells us what changes what.

Conclusion

The survey centers on the relationship between video observations and their underlying dynamic world, not on any single architecture, task, or modality. Under this lens:

Understanding recovers what the video reveals.

Generation constructs video from conditions.

Unification shares information and mechanisms between the two.

World modeling asks how representation state evolves under conditions or actions.

The next phase demands stronger evidence: states that survive occlusion, revisiting, and long-horizon tests; controllable action effects; valid responses to interventions; closed-loop latency meeting control budgets. Only then will the field move from "rendering convincing videos" to "modeling how the world changes."