Major Embodied Intelligence Datasets Shaping Robot Engineering Development

1. ALOHA series: low‑cost dual‑arm interaction datasets

ALOHA (A Low‑cost Open‑source Hardware System for Bimanual Teleoperation) is a dual‑arm teleoperation platform and dataset released by Stanford University and Google DeepMind. The hardware uses inexpensive robotic arms, VR tele‑operation devices and multi‑view cameras, keeping total cost under $5,000 . The dataset focuses on static‑desktop tasks, covering more than 50 task categories (e.g., battery installation, screw tightening, towel folding, zipper closing). Each category provides over 100 demonstration trajectories . Every trajectory records synchronized RGB images (multiple views), depth images, joint angles, end‑effector poses and gripper states—eight core data streams stored in ROS Bag format.

In 2024 ALOHA 2 upgraded the hardware (higher load capacity, improved precision and ergonomics) and added >20 fine‑grained tasks such as electronic component soldering and micro‑assembly, raising the total trajectory count to over 100 k . New annotations include motion smoothness and task‑completion metrics. Mobile ALOHA integrated an omnidirectional mobile base, enabling “mobile‑dual‑arm” tasks (e.g., kitchen cooking, cabinet storage, elevator entry). The mobile dataset adds odometry and LiDAR point clouds for each trajectory.

Models trained on the static ALOHA data achieve >90 % success on desktop tasks via behavior cloning. When training combines static and Mobile ALOHA data, success on mobile tasks rises to ~80 %, exceeding models trained only on static data. The dataset is fully open‑source, compatible with PyTorch, TensorFlow and JAX, and includes a complete code toolchain for loading, preprocessing and visualizing the data.

2. Open X‑Embodiment (OXE): large‑scale cross‑platform interaction dataset

Open X‑Embodiment (OXE) was launched in October 2023 by Google DeepMind together with 21 institutions and 34 labs. It aggregates 60 existing open‑source robot datasets, covering 22 robot platforms (single‑arm, dual‑arm, quadruped, mobile arm, humanoid, etc.), 311 real‑world scenarios, 527 interaction skills and 160 266 distinct tasks, totaling >1 million real‑robot interaction trajectories recorded on physical robots.

All trajectories are stored in the RLDS (Reinforcement Learning Dataset Structure) format, which standardizes timestamps, observations (RGB, depth, point cloud), actions (joint angles, end‑effector pose), reward signals, task descriptions and metadata. RLDS enables efficient parallel loading in TensorFlow, JAX and PyTorch. OXE also provides a full preprocessing pipeline for data cleaning, format conversion, augmentation and trajectory filtering.

Training on OXE enables “general‑purpose” embodied models. RT‑1‑X and RT‑2‑X models trained on OXE achieve a 50 % average task‑success improvement across the 22 robot platforms compared with platform‑specific models, demonstrating cross‑platform skill transfer. OXE 2.0 (2024) adds >10 new robot platforms, >200 k additional trajectories, higher‑quality annotations and multilingual task descriptions.

3. Other prominent embodied interaction datasets

AGIBOT WORLD (2026, Zhiyuan Robotics, China): combines real‑world data and 1:1 digital‑twin simulation data for humanoid robots. Real data cover commercial, household and daily‑operation scenes; simulation data correspond one‑to‑one, supporting a “simulation‑train‑real‑deploy” workflow.

RH20T (Tsinghua University): >200 k humanoid robot trajectories covering 100+ daily tasks (door opening, water pouring, handshakes). Records full‑body joint data, multi‑view vision and tactile sensing, targeting whole‑body control and interaction model training.

RoboMIND (Shanghai AI Lab): >500 k multimodal trajectories, each paired with natural‑language task descriptions, visual scene annotations and intent labels, designed for Vision‑Language‑Action (VLA) model training.

4. Summary of the embodied‑intelligence data ecosystem

The ecosystem follows a three‑layer architecture: real‑world collection, simulation generation and cross‑embodiment fusion. In the real‑world layer, Open X‑Embodiment serves as the global benchmark with >1 M trajectories across 22 platforms, confirming that cross‑embodiment training improves policy generalization. DROID (~76 k trajectories) and BridgeData V2 (~60 k trajectories) provide standardized behavior‑cloning benchmarks for Franka and WidowX platforms.

Domestically, AGIBOT WORLD (2026) offers the first large‑scale dataset covering the full embodiment spectrum for humanoid robots, leveraging exoskeleton tele‑operation and digital‑twin pipelines. The OpenLoong “Baihu” dataset adds multiple robot bodies and end‑effectors, filling national‑standard gaps.

In the simulation and multimodal dimension, RH20T supplies tactile and audio data for contact‑intensive operations, while VTOUCH focuses on dual‑arm coordination and visual‑tactile fusion. Current engineering practice is shifting from fine‑tuning on single datasets to a paradigm of “large‑scale open pre‑training + scenario‑specific private fine‑tuning,” using tele‑operation, simulation synthesis and in‑the‑loop data‑flywheel pipelines to mitigate data scarcity and accelerate VLA model deployment.

Code example

作者：陈之炎
本文
约3200字
，建议阅读
6
分钟
本
本文介绍了主流具身智能数据集，梳理行业数据生态与工程应用价值。