Masked Visual Actions Enable Generalizable Robot Modeling via Pixel Trajectories

Masked Visual Actions (MVA)

MVA encodes robot actions as pixel‑level masks that evolve over time within video frames. The masks preserve object shape, contact, and occlusion, unlike joint‑angle or end‑effector pose representations.

Training data and model

Dataset: ~1,000 DROID demonstrations + 4,000 RoboCasa samples (≈15 h of video), containing both successful and failed trajectories.

Robot region masks are obtained either by segmentation of real video or by rendering from URDF, camera intrinsics, and robot state.

Base model: Wan‑Fun‑Control 2.2, 14 B parameters.

Adaptation: LoRA with rank 256.

Training: ~10,000 steps on eight NVIDIA H200 GPUs, ≈4 days.

Conditional video completion

During training the model receives robot‑trajectory masks as conditioning input. At inference the conditioning can be swapped for an object‑trajectory mask, enabling zero‑shot completion of robot behavior.

Two complementary tasks are expressed as the same conditional video‑completion problem:

Forward prediction : robot‑trajectory → predicted object and environment evolution.

Inverse generation : desired object‑trajectory → completed robot actions.

Quantitative evaluation

DROID test set (unseen real grippers): PSNR 22.79 for MVA, versus 21.02 for skeleton‑based conditions and 20.32 for end‑effector‑pose conditions.

BEHAVIOR dual‑arm robot: PSNR 22.90 for MVA, versus 19.58 and 19.23 for the other two conditions.

Integrating MVA with a Diffusion‑Policy planner improves six RoboCasa tasks by 7–26 percentage points.

Video‑prediction correlation with real‑world success rate: 0.982, indicating reliable low‑cost strategy evaluation.

Real‑robot experiments

Tasks: placing an orange in a bag, stacking blocks, inserting a towel into a bowl, closing a drawer (20 demonstrations each). Generated videos match task progress, though video‑based scores are slightly optimistic.

Zero‑shot CoffeeServeMug task (no training videos): video model + inverse‑kinematics yields 90 % success over 20 trials, outperforming Diffusion Policy (50 %), ACT (80 %) and SmolVLA (85 %). All baselines were trained on 100 demonstrations.

Failure modes and practical considerations

Unrealistic contact, object teleportation, and phantom artifacts appear in fully occluded regions.

Generating many candidate videos incurs high computational cost.

Conditioning on URDF‑rendered trajectories requires accurate robot models and camera calibration.

Cross‑embodiment generalization

Because MVA uses a visual mask as the conditioning modality, different robots (single‑arm, dual‑arm, various grippers) can share the same world model without aligning joint‑space dimensions or low‑level control interfaces.

On unseen real grippers and the BEHAVIOR dual‑arm robot, MVA outperforms skeleton and pose conditions, and can even synthesize additional arms when required.

Links

Paper: https://arxiv.org/abs/2607.19343

GitHub: https://github.com/HadiZayer/masked-visual-actions

Project page: https://masked-visual-actions.github.io/

Code example

本文
约2200字
，建议阅读
5
分钟
本文介绍掩码视觉动作，以像素轨迹实现机器人双向推演与跨具身泛化。