Can Robots Learn Human Moves Directly from AI‑Generated Videos? The GenMimic Breakthrough

Background

Researchers from UC Berkeley, NYU, and Johannes Kepler University propose a framework that enables humanoid robots to reproduce human motions generated by AI video models (e.g., Wan2.1, Sora) without any demonstration data.

Key Contributions

First general framework for executing actions produced by video‑generation models on humanoid robots.

GenMimic reinforcement‑learning strategy that combines a symmetric regularizer with a selectively weighted 3‑D key‑point reward, trained only on motion‑capture data yet robust to noisy synthetic videos.

GenMimicBench, a synthetic human‑motion dataset of 428 videos created with Wan2.1‑VACE‑14B and Cosmos‑Predict2‑14B‑Sample‑GR00T‑Dreams‑GR1.

Extensive validation in simulation and on the 23‑DoF YU‑Tree G1 robot, showing significant improvements over strong baselines.

GenMimicBench Dataset

The dataset contains 428 high‑variance synthetic action sequences covering controlled indoor scenes (217 videos from Wan2.1) and diverse real‑world contexts (211 videos from Cosmos‑Predict2). It spans simple gestures to multi‑step object interactions, providing varied subjects, viewpoints, and environments for robust evaluation.

Two‑Stage Reconstruction Pipeline

Stage 1 – Pixel to 4D humanoid reconstruction: An off‑the‑shelf human‑reconstruction model extracts per‑frame global pose and SMPL parameters from the generated video. Because the SMPL mesh does not match the robot’s morphology, the SMPL trajectory is redirected into the robot’s joint space, yielding 3‑D key‑points in robot coordinates.

Stage 2 – 4D humanoid to robot actions: The policy consumes the 3‑D key‑points and proprioceptive data, outputting physically feasible joint‑angle targets that are tracked by a PD controller.

GenMimic Policy

The policy is trained with PPO augmented by two novel components:

Weighted Tracking: A per‑key‑point error term is weighted so that critical points (e.g., end‑effector) dominate the reward, reducing the impact of noisy lower‑body points.

R_{track}=\sum_{i}\omega_i\|k_i^{pred}-k_i^{gt}\|_2

Symmetry Loss: An auxiliary loss encourages left‑right key‑point symmetry, exploiting the inherent bilateral symmetry of the human body.

L_{sym}=\lambda_{sym}\sum_{j}\|k_{j}^{L}-k_{j}^{R}\|_2

Experiments

Simulation

Training was performed in IsaacGym with over 1.5 billion samples on four NVIDIA RTX 4090 GPUs. Evaluation on GenMimicBench shows the GenMimic student and teacher models outperform baselines (GMT, TWIST, BeyondMimic) in success rate (SR) and mean per‑key‑point error (MPKPE‑NT).

Real‑World

The policy was deployed on a 23‑DoF YU‑Tree G1 robot using a single NVIDIA RTX 4060 mobile GPU. Out of 43 tested actions, the robot successfully reproduced a wide range of upper‑body motions (waving, pointing, stretching) and some multi‑step sequences. Failures occurred mainly in lower‑body locomotion and complex turn‑and‑step combos, likely due to inaccurate or physically infeasible video cues.

Resources

Paper: https://arxiv.org/abs/2512.05094v1

Project website: https://genmimic.github.io/