EgoSteer: System for Dual Dexterous Hands Using Human Data and World Models

Overview

EgoSteer is a full‑stack system for controlling a pair of dexterous robot hands. It combines a large‑scale data pipeline, a vision‑language‑action (VLA) model, and a robot control stack, all of which are open‑source (paper: https://egosteer.github.io/EgoSteer.pdf, project: https://egosteer.github.io/#/). The system can follow open‑ended natural‑language instructions and generalize to unseen scenarios.

Data Pipeline – EgoSmith

The EgoSmith pipeline converts noisy first‑person videos into high‑quality VLA training data through four automated stages:

Stage 1 – Pre‑filtering: Heuristic rules discard frames with large optical flow (indicating walking) and use YOLO to remove frames with severe hand occlusion or extraneous people.

Stage 2 – 4D Motion Estimation: DPVO provides camera tracking and key‑frame depth; Any4D predicts per‑frame metric depth. After scale alignment, hand motions are transformed from camera to world coordinates.

Stage 3 – Multi‑level Language Annotation: Qwen3.5‑VL‑Plus filters out non‑interactive clips and generates five levels of textual instructions (L1–L5), ranging from atomic verb‑object pairs to detailed step‑by‑step descriptions.

Stage 4 – Post‑filtering: Multi‑level quality control removes outlier episodes, chunks, and frames based on camera translation distribution, wrist pose consistency, and frame‑wise motion deltas.

The final dataset spans 9.6 K hours of video, 2.09 M clips, 10.4 B frames, covering 8 969 object nouns and 623 action verbs, drawn from roughly 11.6 K hours of publicly available first‑person footage.

Model Architecture

EgoSteer’s VLA model consists of three cooperating components that share a visual‑language backbone (Qwen3‑VL, 2 B parameters, causal attention):

Action Expert: A DiT‑based module (~300 M parameters) that generates action chunks via flow‑matching and attends jointly to its own tokens and the backbone representation.

World‑Model Expert: A lightweight 4‑layer Transformer (~70 M parameters) that predicts future DINOv3 visual features of the next frame given the current action and camera motion, providing a “future imagination” signal during training.

Unified Action Space: Wrist pose is encoded as 3‑D position plus 6‑D rotation; each finger tip is represented relative to the wrist, yielding a 48‑dimensional vector for both hands.

The world‑model expert is only active during training; it is removed at inference time, incurring no extra computational cost.

Real‑Time Chunking (RTC)

To avoid jitter during inference, EgoSteer trains with Real‑Time Chunking: the model receives a clean action prefix with random delay and learns to denoise the subsequent actions, making the deployed system robust to inference latency.

Human‑in‑the‑Loop Correction

The system supports seamless human takeover using a “relative motion mapping” strategy. When an operator presses a foot pedal, the current robot end‑effector pose and the human hand pose are recorded as baselines. Subsequent human motions are applied as relative offsets to the robot’s state, enabling smooth hand‑over without abrupt jumps. The success rate of this hand‑over exceeds 85 %.

Experimental Evaluation

EgoSteer was evaluated on 40 manipulation tasks across seven categories (e.g., pick‑and‑place, non‑grasp pushing, object redirection, bimanual collaboration, dense contact). Each task was run ten times in random cluttered scenes. Results:

Overall average success rate: 75 % .

Difficulty tier 1 (seen tasks): stable performance.

Tier 2 (compositional generalization): 65 % success.

Tier 3 (unseen semantic tasks): 62 % success.

Scaling experiments showed that increasing pre‑training data improves performance: with 9.6 K h of human video the success rate reaches 60 % , versus 30 % for a model trained from scratch, a 30‑point gain attributed entirely to pre‑training data.

Few‑Shot Learning

On the RealMan robot, only 120 demonstrations were needed for the 18‑step, 40‑second box‑folding task, achieving 75 % success. On the AgiBot G1 platform, 200 demonstrations sufficed for a 9‑step cake‑box task, reaching 83 % success. Competing methods (Diffusion Policy, IMLE, and a non‑pre‑trained baseline) achieved 0 % success under the same data budget.

Comparison with Baselines

Against contemporaneous works on a set of ten easy tasks, EgoSteer attained an average success of 74 % , while Being‑H0.5 achieved 39 % and π0.5 achieved 22 % . The paper attributes the gap to differences in pre‑training, action representation consistency, input resolution, and deployment optimizations.

Conclusion

EgoSteer demonstrates that a carefully engineered data pipeline, a VLA model augmented with a transient world‑model expert, and a unified action representation can enable dual dexterous hands to execute a wide range of language‑driven tasks with strong generalization. The open‑source release provides a reproducible baseline for future embodied‑AI research.