Embodied AI's GPT Moment? ZJU & Unitree Open-Source 5M Robot Pretraining Dataset
Zhejiang University and Unitree Robotics release Movens-5M, a 5-million-sample robot pretraining dataset with 10,000+ hours of simulated verified actions, and train Movens-Zero, a multimodal model generating full-body motions from text, speech, music, or video, demonstrating scaling benefits on Unitree G1.
Movens-5M Dataset and Movens-Zero Model
The Zhejiang University and Unitree Robotics Embodied Intelligence Joint Laboratory has open-sourced Movens-5M , a large-scale pretraining dataset containing 5 million robot actions (over 10,000 hours) derived from massive internet human videos. Each action is verified by simulation execution on the Unitree G1 humanoid robot. The dataset covers 297 fine-grained action types across 16 scenario categories including household chores, fitness, and outdoor leisure.
Crucially, the dataset achieves multimodal alignment : text instructions, video demonstrations, speech, and music are paired with robot motions, and time-segmented, body-part-level textual descriptions are provided. This enables models to learn how each input modality maps to specific robot actions.
Data Processing Pipeline: Five Automated Stages
To scale from individual examples to millions of samples, the team built an automated pipeline with five stages:
Clip Generation : Long videos are split into continuous clips along shot boundaries, preserving source metadata.
Quality Filtering : Clips are screened for visual quality, human visibility, and motion information richness; low-quality material is discarded.
Semantic Annotation : Each clip receives scene and action descriptions, time-segmented body-part motion narratives, and associated text, speech, and music information.
Motion Recovery : 3D human pose is estimated from video frames, capturing how the body moves over time.
Physical Refinement : Motion jitter and instability are corrected; motions are retargeted to the G1's kinematic structure (different proportions, joint limits) and verified in simulation for stable execution.
The pipeline yielded 15 million human motions and 5 million robot motions , totalling over 10,000 hours of behavioral data with detailed textual annotations.
From Human Video to Robot Action: A Cleaning Example
A 7.25-second door-wiping video illustrates the process. Four synchronized views show: (1) original video, (2) recovered 3D human motion, (3) retargeted G1 reference motion, (4) G1 simulation execution. The motion is divided into three phases with body-part descriptions:
0–4 s : Person bends forward, left hand wipes lower door panel with green cloth, right arm bent behind body.
4–6 s : Person gradually straightens up, left arm rises with wiping position.
6–7.25 s : Both arms reach upward toward higher door areas.
Object-interaction examples include dumbbell curls (bicep curls and bent-over rows) and barbell deadlifts (forward lean, controlled lowering, standing up), showing coordinated arm, leg, and trunk movement.
Movens-Zero: One Model for All Modalities
Trained on the paired data, Movens-Zero uses an understanding module to process text, speech, music, or video, and a single motion expert (generation module) that combines the understood input with the robot's current state to plan full-body motions. The robot controller then sends joint commands.
Demonstrated Capabilities
Text-to-motion : Input "Dance some hip-hop" generates a hip-hop dance without a reference video.
Speech-to-motion : Verbal command "Salute with your left hand" produces the corresponding salute.
Music-driven motion : The model generates stepping and upper-body movements synchronized to rhythm and style.
Video-to-motion : Given a video demonstration (internet clip or live capture), the model outputs a robot-feasible motion.
Real-Time Closed-Loop Control
During live interaction, Movens-Zero generates short motion chunks. The robot executes the first part while the model retains the tail as the next chunk's prefix, then conditions on the latest input and robot state to produce the continuation. This creates a real-time perception–generation–execution loop that adapts to changing human instructions or demonstrations.
Scaling Experiments: Data and Model Size
The team conducted systematic scaling experiments to test whether more data and larger motion experts yield sustained improvements.
Data Scaling (Fixed XL Motion Expert: 910.5M Parameters)
Training data was increased from 1% to 100% of Movens-5M (deduplicated motion duration from 122 to 12,231 hours). Key results:
Text–motion matching (R@3) : 12.72% → 62.57% (higher is better).
Video motion error (MPJPE) : 132.82 mm → 40.90 mm (69.2% reduction; lower is better).
Speech–motion matching : 13.98% → 74.30%.
Music beat alignment (BeatAlign) : 0.23 → 0.61.
Tracking Score : 0.21 → 0.72 (overall execution fidelity).
Gains persisted from 30% to 100% data: R@3 rose another 20.77 percentage points , MPJPE dropped a further 53.5% .
Model Scaling (Full Dataset, Four Motion Expert Sizes)
Understanding modules fixed; only motion expert size varied:
S : 64.2M params
M : 187.9M params
L : 500.1M params
XL : 910.5M params
Results:
Validation Flow Matching Loss : 0.487 → 0.159 (67.4% reduction).
Tracking Score : 0.18 → 0.72.
All modality-specific metrics (text, speech, music, video) improved consistently with model size.
Interaction: Larger Models Benefit More from Additional Data
When data increased from 10% to 100%, the relative improvement in R@3 was 62.5% for S but 151.1% for XL . The same trend held across all five other metrics: larger motion experts extract more value from the same data increments.
Conclusion
The experiments demonstrate that data scaling and model scaling are complementary : more data teaches richer motion repertoires, while larger motion experts can more fully exploit that data. This synergy suggests continued gains from further scaling.
Resources
Dataset: huggingface.co/datasets/wendell0218/Movens-5M Project page: movens-zero.github.io Paper:
arxiv.org/abs/2610.41877Signed-in readers can open the original source through BestHub's protected redirect.
This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactand we will review it promptly.
How this landed with the community
Was this worth your time?
0 Comments
Thoughtful readers leave field notes, pushback, and hard-won operational detail here.
