EgoWild2Dex: Robots Master Complex Tasks from 538 Hours of 'Dirty' Human Egocentric Data

Background: The Cost of Robot Data

Robot learning has long faced a practical bottleneck: collecting real robot demonstrations is expensive, especially for complex tasks requiring bimanual coordination, tool use, fine contact, and long-horizon sequences. Humans, by contrast, perform massive amounts of manipulation daily in homes, factories, pharmacies, hospitals, supermarkets, and parcel stations. These everyday actions constitute a vast, readily available source of operational experience.

EgoWild Dataset: Large-Scale "Dirty" Human Egocentric Data

The team collected EgoWild , a dataset of 538.9 hours of natural human first-person (egocentric) video spanning homes, factories, pharmacies, hospitals, supermarkets, and parcel stations. The data is deliberately "dirty" — it retains camera shake (average cumulative head rotation 15.93° per second ), complex backgrounds, long-tail object distributions (average 34.78 object instances per frame ), and natural human motion. It contains 179,000 operation clips , 126,000 independent task descriptions , and 1,282 object categories . Annotations include full-body pose, hand joints, and a three-level hierarchy: atomic actions, action commands, and skill commands. Specialized processing addresses hand-action recognition and human sliding steps.

Method: EgoWild2Dex Framework

GeoFormer: Geometric View Alignment

The first obstacle is the viewpoint gap : human head-mounted cameras move dynamically while robot cameras are relatively stable. EgoWild2Dex introduces GeoFormer (Geometric Transformer) , a lightweight differentiable geometric transformation module. GeoFormer learns the geometric relationship between human egocentric views and robot views and dynamically warps human observations to match the robot perspective. Compared to traditional Project + Inpaint pipelines that rely on precise camera calibration, 3D reprojection, and image inpainting, GeoFormer reduces dependence on exact calibration and achieves 21.9× faster processing with better visual alignment quality.

Unified Action Space via Retargeting and Inverse Kinematics

Three data sources are mapped to a common robot-native action space:

Egocentric human data from natural environments (Ego)

Glove-based teleoperation demonstrations from robot perspective (Glove)

Real robot demonstrations (Robot)

Action retargeting and inverse kinematics convert all three into unified robot actions, enabling joint policy training.

Progressive Human-Robot Co-Training

Training proceeds in three stages:

Open-Set Pre-training (Human-to-Robot Learning) : Large-scale in-the-wild egocentric data pre-trains the VLA model, exposing it to diverse scenes, objects, and manipulation patterns to learn general visual and manipulation priors.

Human-Robot Co-training : Task-relevant egocentric data, glove demonstrations, and a small amount of real robot data are added. The unified action space links broad human experience to specific robot tasks.

Robot-Domain Refinement : A few real robot demonstrations and recovery trajectories fine-tune the policy to adapt to real-world physics, contact dynamics, and execution errors. Each downstream task uses less than 1 hour of real robot data.

Experiments: Real-World Evaluation

Three complex long-horizon bimanual tasks were tested on a Songling Piper dual-arm system:

Box opening with a utility knife : cut tape, open lid, retrieve object.

Hot-glue gun figurine fixation : coordinated bimanual operation — align, apply glue, grasp, fix.

Ice drink preparation : scoop ice, pour ice, hold cup, align spout, catch water.

The model achieved an average task success rate of 96.7% . It demonstrated robustness to external disturbances (finger jamming, human replacing objects) by adjusting actions and continuing execution. Zero-shot generalization to unseen objects in the box-opening task yielded a 33.3% success rate . The framework was also transferred from the Piper arm to a Tianji Marvin Pro arm, showing cross-embodiment feasibility.

Conclusion and Outlook

EgoWild2Dex demonstrates that uncurated, "dirty" human egocentric data — previously considered too noisy for robot training — can be effectively leveraged through geometric view alignment (GeoFormer), unified action mapping, and progressive co-training. This expands the data sources for robot learning beyond expensive real-robot demonstrations. While viewpoint, embodiment, and dynamics gaps remain, the approach points toward robots that continuously adapt and evolve from human daily experience in complex real worlds.

Paper: arXiv:2609.23755 (https://arxiv.org/pdf/2609.23755) | Project page: https://mmlab.hk/egowild2dex (dataset, model, and code to be released).