Rendering Robot Actions as Video for Cross‑Embodiment Bidirectional Inference

In model‑driven robot planning, candidate actions are first generated and then evaluated by a video world model that predicts future frames given the current image and numeric action data (joint angles, end‑effector pose, gripper state). Establishing a mapping between these numeric actions and visual observations requires additional learning, and the data distribution changes when a different robot is used.

To address this mismatch, the authors propose Masked Visual Actions , which render candidate robot motions as pixel‑level masks and let a video model fill in the resulting environmental response. This enables two bidirectional capabilities:

Given a robot motion, predict how the environment will change.

Given a desired object motion, generate plausible robot actions.

The method was introduced in the paper Masked Visual Actions for Unified World Modeling (Hadi Alzayer et al., arXiv 2026.07.19343) with contributions from Fei‑Fei Li and others.

Training data are constructed in two ways: (1) directly segmenting the robot from real videos, preserving authentic visual information; (2) rendering robot trajectories from URDF models and camera parameters, which allows arbitrary motion inputs at inference time. The model is fine‑tuned via LoRA on the 14B Wan‑Fun‑Control 2.2 foundation model using roughly 15 hours of robot interaction data (≈1 000 DROID demonstrations and 4 000 RoboCasa samples, trained for ~10 000 steps on eight H200 GPUs over four days).

Experimental results show that integrating the video model into planning improves success rates on tasks such as closing a microwave, opening a drawer, or operating a dishwasher by 7–26 percentage points. In the RoboCasa benchmark, the video‑augmented strategy correlates with real‑simulation outcomes at 0.982. For the CoffeeServeMug task, the full pipeline (video model + inverse dynamics) achieves 90 % success, surpassing Diffusion Policy, ACT, and SmolVLA.

Compared with the earlier BridgeV2W approach, which injects robot masks into a pretrained video model, Masked Visual Actions abstracts the problem to a fill‑in‑the‑blank formulation: given any subset of entities in an interaction video, predict the missing parts. This yields stronger cross‑embodiment generalization; when tested on unseen grippers or a dual‑arm robot, the full pixel mask maintains robot shape and motion more stably than end‑effector points or skeletal representations.

The authors note limitations: the method relies on a large video foundation model and high‑end hardware, lacks strict causal and physical constraints (leading to occasional unrealistic motions such as object teleportation), and requires an additional inverse‑dynamics module to convert generated videos into executable commands. Consequently, while useful for low‑cost early‑stage strategy screening, it is not yet a replacement for simulators or real‑world testing.