Skip to content
DexRoam: Learning Mobile Bimanual Dexterous Manipulation from Human Demos

DexRoam: Learning Mobile Bimanual Dexterous Manipulation from Human Demos

DexRoam aligns human whole-body demonstrations to robot actions, enabling mobile bimanual dexterous manipulation learning from egocentric human videos and cutting robot teleoperation data needs by half.

Data Party THU
Data Party THU
Data Party THU

Problem Background

Complex mobile bimanual dexterous manipulation requires continuous coordination of base, torso, dual arms, head, and dexterous hands. Collecting high-quality robot teleoperation demonstrations for such tasks is costly and often requires multiple operators. Human daily activities naturally contain rich whole-body coordination experience, raising the question: can robots learn directly from human demonstrations?

DexRoam System Overview

Researchers from HKUST, Beijing Academy of AI, and Peking University propose DexRoam, a complete pipeline from egocentric human demonstration collection, whole-body motion alignment, to robot policy learning and real-world deployment. The work is accepted at CoRL 2026.

Data Collection

Human Demonstration Collection

A portable egocentric whole-body capture system uses only a Meta Quest 3 headset and a head-mounted ZED Mini stereo camera. It synchronously records first-person video, full-body joint motion, and hand keypoints without external motion capture, environment markers, or additional body trackers. The camera feed is displayed in the headset, matching the visual observation during policy training.

Robot Teleoperation System

A single-operator whole-body teleoperation system uses a VR headset, controllers, and data gloves to simultaneously control the robot base, torso, dual arms, head, and dexterous hands. Both systems provide human and robot demonstrations for joint learning.

Three-Stage Human-Robot Motion Alignment

To bridge embodiment gaps while preserving continuous whole-body coordination, DexRoam designs three alignment stages:

1. Embodiment Alignment

Establishes motion mapping between human and robot. Using the human pelvis as reference, human base, torso, dual-arm, and head motions are mapped to corresponding robot parts. Hand retargeting generates dexterous hand commands, preserving relative whole-body motion relationships.

2. Semantic Alignment

Unifies action representations. Base, torso, and dual-arm motions are expressed as relative action representations, reducing dependence on human absolute posture (height, arm length, initial pose) so both demonstration types share consistent control meaning.

3. Temporal Alignment

Matches execution rhythms. Human trajectories are resampled according to task progress to match the robot demonstration time scale while preserving original motion trajectories, providing consistent temporal supervision.

Policy Training and Experiments

After alignment, human and robot demonstrations share a unified observation-action interface and can jointly train Vision-Language-Action (VLA) policies without modifying model architecture for human data. Two VLA models are used: GR00T N1.7 and π₀.₅. Three training strategies are explored: joint human-robot training, human pretraining then robot fine-tuning, and human pretraining then joint fine-tuning.

Results

Alignment Effectiveness

PCA visualization shows significant distribution overlap after full alignment. Quantitatively, MMD and SWD metrics decrease by 41.7% and 31.1% on average. Aligned human trajectories can be successfully replayed on the real robot.

Real-World Policy Performance

Across five tasks and both VLA models, all three strategies incorporating human demonstrations improve performance. The best strategy (human pretraining + joint fine-tuning) raises average success rates: GR00T N1.7 from 29% to 56%, π₀.₅ from 32% to 57%.

Human demonstrations also introduce new manipulation behaviors. For example, picking up a Pringles can: robot-only policies mainly grasp from the front and often knock it over when orientation changes; with human data, the robot learns side grasping — a behavior present only in human demonstrations.

Data Efficiency

Keeping human demonstration count fixed while halving robot demonstrations yields performance close to the full robot-data baseline on both models. This suggests aligned human demonstrations can replace ~50% of costly robot teleoperation data. Few-shot experiments further show rapid improvement with minimal robot data.

Conclusion

DexRoam demonstrates that human data can cross embodiment gaps to become direct action supervision for complex whole-body manipulation. The value lies not only in visual and task information but in fine-grained, interrelated whole-body motion experience. As such experience transfers effectively, learning complex mobile dexterous manipulation can reduce dependence on expensive whole-body teleoperation data, opening possibilities for larger-scale robot skill learning.

Code and data collection/alignment tools are open-sourced: https://github.com/zhourui9813/DexRoam and policy training code: https://github.com/zhourui9813/DexRoam-Policy-Training. Project page: https://dexroam.github.io/. Paper: https://arxiv.org/abs/2609.35761.

Original Source

Signed-in readers can open the original source through BestHub's protected redirect.

Sign in to view source
Republication Notice

This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactadmin@besthub.devand we will review it promptly.

imitation learninghuman-robot interactiondexterous manipulationrobot learningmobile manipulationCoRL 2026bimanual manipulationDexRoam
Reader feedback

How this landed with the community

Sign in to like

Rate this article

Was this worth your time?

Sign in to rate
Discussion

0 Comments

Thoughtful readers leave field notes, pushback, and hard-won operational detail here.