WOVEN: World Modeling as Cross-Task Capability with 2K Samples Boosting 22 Benchmarks

What Is World Modeling?

The paper narrows the definition of world modeling to its core: state transition. Given a current state s, an action a, and a next state s', a world model estimates p(s' | s, a). Under this definition, world modeling becomes a precisely controllable and measurable capability: inferring the missing element of a transition triple (s, a, s'). Spatial, embodied, physical, and temporal tasks unify as different queries on the same triple — predicting s' from s and a (forward), inferring a from s and s' (inverse), counterfactual reasoning by changing a, and temporal ordering of intermediate states. This aligns with cognitive science's forward/inverse models and infants' staged acquisition of core physical knowledge (gravity, support, solidity, etc.).

Making World Modeling a Controllable Experimental Object

To intervene on individual transition elements, the team uses a video generation model as a transition engine: given an initial frame and an action description, it generates the resulting trajectory. From the same start frame, multiple actions yield counterfactual trajectories with ground-truth controls. Scene, action, and reasoning operation become independent experimental variables (20 scene categories, 5 action types, 8 reasoning operations). Each question includes error-type labels (e.g., left-right flip, gravity violation, object orientation perturbation), enabling fine-grained error analysis. This constitutes the WOVEN benchmark: 36K questions for training and evaluation.

What Current Models Lack

The team evaluated 38 frontier models. The strongest, GPT-5.4, achieves 65.8% accuracy versus human 92.3%. The gap pattern is consistent across model families: temporal adjacent judgment is weakest at 25–28% (near random), while temporal ordering reaches 63%. Appearance perturbations (color change) barely affect performance, but geometric perturbations (position/orientation change) drop accuracy to ~30%. Scaling alone does not close this gap: on Qwen2.5-VL from 3B to 72B, appearance robustness improves 31 points per 10× parameters, geometric robustness only 9 points. A 7B model scores 50.5% under appearance perturbation but 23.5% under geometric perturbation — effectively no learning. Even optimal read-out from video priors (V-JEPA 2, a video generation model) tops at 47.7%.

Models acquire physical principles in the same order as infants: gravity and inertia first, then cohesion, permanence, containment, then support and solidity, finally collision — consistent across families. Collision remains at 31.4% average across 38 models. Open models confuse rotation direction: they know rotation occurs but pick the mirrored direction over 50% of the time. The shared failure distribution indicates a training problem, not model-specific quirks.

Learning World Modeling Efficiently and Transferring to Downstream Tasks

World modeling can be learned independently. Post-training (SFT + RL) on WOVEN lifts Qwen2.5-VL-3B from 26.4% to 89.3% in-domain, and 88.2% on unseen scenes (human 92.3%). Same holds for 7B, 32B, and Gemma 3. Learning is efficient: ~7 hours of generated video post-training on Qwen2.5-4B yields +6.7 on 3DSRBench, surpassing Orca-4B's +4.0 from 125k hours of real-video mid-training.

Full WOVEN training improves 12 external benchmarks spanning physical understanding, temporal reasoning, embodied manipulation, and spatial reasoning. When taking the union of transfer gains from 11 controlled subsets (each only 2,000 samples), the number of improved external benchmarks rises to 22, with zero-shot gains up to 27.3 points.

World Modeling as a Shared Underlying Capability

Two conditions must hold for a "shared underlying capability" claim. First, diverse supervision sources must teach the same capability. All 11 controlled subsets improve untrained action types (e.g., camera-motion-only data boosts object manipulation +39.8 and navigation +26.9). After training, forward and inverse questions on the same transition become "same right, same wrong"; inverse prediction from the model's own forward prediction rises from 19% to 45–94%. The model learns a unified state-action-outcome mapping, not separate tricks.

Second, downstream gains must stem from this capability, not superficial stacking. Transfer curves of the 11 subsets across 26 benchmarks are pairwise positively correlated; a subset's world-modeling improvement correlates with its average external gain (r = 0.86). Four static perception and general video QA benchmarks never improve, ruling out overall model strengthening. A causal control: a "virtual world" with fixed but incorrect action-outcome mappings (e.g., turning left always shows right-side objects) is learned perfectly (85–92%), yet downstream benchmarks show zero gain — proving downstream improvement requires learning correct transition laws.

Direct causal intervention: replacing 30–50% of downstream task training data with WOVEN data maintains downstream accuracy and outperforms simply deleting that data (so not due to data saturation). Transfer supervision substitutes for what the downstream task would have learned from its own data.

Training Recipe: Reasoning Operation Matters More Than Action Similarity

Intuition suggests selecting supervision similar in scene, action, or domain. Experiments show otherwise. After removing common transfer components, subsets sharing the same reasoning operation correlate at +0.32, while subsets sharing only action type correlate at -0.22 (indistinguishable from unrelated subsets at -0.23). Thus, "what reasoning is performed on the transition tuple" determines transfer, not "what action was taken."

Additional findings: for robustness, prioritize supervision with larger visual state changes; temporal and agent-driven tasks need direct supervision; active (agent-driven) and passive (physics-driven) actions show no significant cross-transfer; static perception cannot benefit from state-transition supervision and requires its own signal.

Prospective validation: the team composed training mixtures per the recipe, then tested on 7 unseen benchmarks (MMSI-Bench, VLM4D, PhysBench, SPAR-Bench, MVBench, IntPhys 2, CameraBench) with 8 random seeds. All five comparison directions matched recipe predictions: on agent-action questions, recipe mixtures significantly outperform alternatives (p ≤ 0.03); on passive-physics questions, passive-physics-heavy mixtures win as predicted. Predictive power on unseen tasks outweighs post-hoc explanation.

Takeaways and Limitations

The paper's methodology: disentangle variables for interventional infrastructure; use cross-domain tests, wrong-rule controls, and causal interventions to rule out spurious correlations; isolate "underlying capability" from "effect stacking" with rigorous causal and statistical analysis; prospectively validate the derived training recipe on held-out tasks. Each step carries its own evidence; conclusions do not rest on a single metric.

For training systems, world modeling should be organized by reasoning operation, not by task, domain, action, or scene type — enabling more efficient training of more intelligent systems. Transfer is not unconditional: temporal tasks, agent-driven tasks, and static perception are boundaries where transfer supervision helps little or not at all. The recipe provides concrete data-selection rules, not a vague "synthetic data helps" claim.

Paper: https://arxiv.org/abs/2610.12417

Project page: https://woven-ai.github.io

Code: https://github.com/mll-lab-nu/WOVEN

Data: https://huggingface.co/datasets/MLL-Lab/WOVEN