10,000‑Hour Human Data Powers the First World‑Action Model for Humanoids

Challenges in Humanoid‑Robot Learning

Collecting high‑quality robot demonstration data is expensive and risky because it requires synchronized first‑person video, proprioceptive sensing, and full‑body command recording. Pixel‑level video prediction models are computationally heavy and produce noisy predictions under rapid robot motion and camera shake. Existing pipelines often model arm manipulation and locomotion separately, limiting coordinated whole‑body control.

Being‑M0.7: Implicit Latent World‑Action Model

Being‑M0.7 is an implicit latent world‑action model (Latent WAM) for full‑body mobile operation of humanoid robots. The model is pretrained on more than 10,000 hours of mixed‑modality human data, including first‑person video, video‑motion paired clips, and pure motion sequences, then adapted with a small set of real‑robot demonstrations.

Vision‑Motion MoT Architecture

The backbone is a Vision‑Motion MoT (Mixture‑of‑Transformers) that keeps separate visual and motion streams while allowing cross‑modal interaction through shared attention. Visual and motion modalities retain dedicated projection layers; shared multi‑modal attention enables information exchange when both modalities are present.

Training accommodates three data types:

Video‑motion paired data: joint loss on future visual states and motion trajectories.

Pure video data: loss applied only to the visual branch.

Pure motion data: loss applied only to the motion branch.

From a probabilistic view, paired data model the joint visual‑motion distribution, while single‑modality data provide marginal constraints, allowing incomplete modalities to be incorporated without separate models.

Unified Motion Representation

Human and robot motions are converted to a head‑rooted representation that includes both hands and feet. The representation is standardized by aligning coordinate frames and removing initial orientation, reducing distribution gaps between datasets. A compact version retains only head, hands, and feet, preserving key interaction information while bridging human‑robot morphology differences.

Pretraining Objective

Visual frames are encoded into a latent space; the unified motion representation is concatenated. Training uses a flow‑matching loss to jointly predict future visual states and motion trajectories from a short history of visual‑motion observations and task instructions.

Real‑Robot Fine‑Tuning with Action Expert

During fine‑tuning, a lightweight Action Expert reads the latent model’s hidden states, combines them with current visual observations and proprioception, and generates executable motion blocks. The expert maps high‑level latent plans to the robot’s 29‑DOF control commands.

Inference: Low‑Frequency Planning & High‑Frequency Control

At inference time the model generates low‑frequency world‑level video‑motion plans, stores intermediate hidden states in a KV cache, and uses the Action Expert to produce high‑frequency motion commands that are continuously corrected with fresh visual and proprioceptive feedback.

Real‑Robot Demonstrations

Four tasks evaluate full‑body coordination:

Fish‑netting : robot approaches a water tank and uses a handheld net to capture a toy fish. Success: 3/5 attempts (Being‑M0.7) vs 2/5 (GR00T‑N1.6) vs 1/5 (baseline).

Mirror retrieval : robot infers the 3D position of an object hidden inside a box using a mirror reflection. Success: 4/10 total (3/5 at 0.5 m, 1/5 at 1 m) vs 1/10 for GR00T.

Moving‑object transport : robot walks to a table, transfers a baguette between baskets, picks up a bouquet, and walks away, requiring continuous switching between locomotion and manipulation.

Box carrying with obstacle avoidance : robot carries a box, detects an obstacle, adjusts body orientation, and sidesteps through a narrow gap while maintaining balance and load stability.

These demos show that the robot continuously generates and refines whole‑body actions based on current observations, real‑time feedback, and predicted future states.

Data Recipe and Collection Pipeline

Pretraining data combine public datasets (Ego4D, Xperience, Nymeria, Bones‑SEED, SnapMoGen, HumanML3D, Lafan1) and internal collections, totaling over 10,000 hours of mixed‑modality recordings.

Real‑robot data are collected with a PICO VR system: the operator wears a headset, ankle trackers, and hand controllers; the VR system estimates full‑body pose, converts it to 29‑DOF robot commands, and simultaneously records first‑person RGB, proprioception, and command streams.

Scalability and Future Directions

The Vision‑Motion MoT paradigm decouples data requirements from strict visual‑motion pairing, allowing future scaling with additional video‑only or motion‑only sources. By first learning visual‑motion priors from abundant human behavior and then transferring them to robot‑specific control, the approach demonstrates a sustainable path for expanding embodied intelligence beyond the bottleneck of scarce robot demonstrations.

Paper: https://research.beingbeyond.com/being-m07/being-m07.pdf

Project page: https://research.beingbeyond.com/being-m07