Masked Visual Actions: Controlling Robots with Only 15 Hours of Video

Overview

Fei‑Fei Li’s Stanford team, together with researchers from the University of Maryland and Harvard, introduced Masked Visual Actions , a method that encodes robot actions as pixel‑level spatiotemporal masks in video. By adapting a 14‑billion‑parameter pretrained video model (Wan‑Fun‑Control 2.2) with LoRA (rank 256), the system can predict the consequences of a given action and generate robot behavior from visual conditions.

Training Data

The model is trained on roughly 15 hours of robot video, comprising 1,000 demonstrations from the DROID dataset and 4,000 samples from RoboCasa. During training only robot motion conditions are provided; at inference the condition can be swapped to object trajectories, enabling zero‑shot completion of robot behavior.

Model Details

The base model, Wan‑Fun‑Control 2.2 (14 B parameters), is fine‑tuned for 10,000 steps on eight H200 GPUs (≈4 days). The masked visual action representation keeps the spatiotemporal mask of the entity while rendering the rest of the frame as a gray background, preserving shape, position, scale, and contact information.

Cross‑Embodiment Generalization

Because the representation is visual rather than joint‑space based, the same model works for single‑arm, dual‑arm, and different‑gripper robots without sharing joint dimensions. On unseen real grippers the mask‑visual‑action condition achieves a PSNR of 22.79 , outperforming skeleton (21.02) and end‑effector pose (20.32). On the BEHAVIOR dual‑arm robot the PSNR reaches 22.90 , higher than the other two conditions (19.58 and 19.23).

Dual Usage: Forward Prediction and Inverse Generation

With the same model, providing a robot trajectory as input predicts object and environment changes (forward prediction), while supplying a target object trajectory causes the model to synthesize the corresponding robot motion (inverse generation). The generated video is later converted to low‑level commands by an inverse‑kinematics model.

Planning and Evaluation

Combined with Diffusion Policy and Gemini 3.1 Pro Preview, the world model scores candidate trajectories on task completion, contact realism, and physical plausibility. On six RoboCasa tasks the success rate improves by 7–26 percentage points. Video‑based evaluation correlates strongly with real‑world success (Pearson 0.982), showing that generated videos can reliably compare strategies.

Real‑World Experiments

Experiments on tasks such as orange‑in‑bag, block stacking, towel‑in‑bowl, and drawer closing (20 demonstrations each) show a 90 % success rate after inverse‑kinematics conversion, surpassing Diffusion Policy (50 %), ACT (80 %) and SmolVLA (85 %). Failure cases include false contact, object teleportation, and artifacts when unseen regions are rendered.

Limitations and Future Work

Video generation remains costly, and rendering conditions requires URDF models and camera calibration. Occasionally the model produces unrealistic contact or objects moving without the gripper. As video quality, inference speed, and inverse‑kinematics models improve, the approach could scale to more robot morphologies and more complex manipulation tasks.

Resources

Paper: https://arxiv.org/abs/2607.19343<br/>Code: https://github.com/HadiZayer/masked-visual-actions<br/>Project page: https://masked-visual-actions.github.io/