What Data Should Robots Learn From? A Five‑Layer Data Pyramid Framework

Multimodal foundation models achieve impressive perception and reasoning by training on internet‑scale image‑language data, but embodied intelligence cannot rely on the same shortcut; robots must learn from data that jointly captures observations, states, actions, and physical consequences.

The authors introduce a "Data Pyramid for Embodied Manipulation" that organizes existing data sources into five complementary tiers: real‑robot data, UMI‑style data, ego‑centric/self‑center and external‑view data, simulation data, and generic vision‑language data. The pyramid is arranged along two axes—scalability (cost, hardware, human effort) and robot‑alignment (how directly the data supports robot learning).

Beyond these axes, each tier is evaluated on four supplemental dimensions: quality (reliability of trajectories, annotations, synchronization), diversity (coverage of tasks, objects, scenes, viewpoints, modalities), reusability (ability to transfer across tasks, environments, embodiments), and physical fidelity (accuracy of contact, friction, sensor noise, control latency, and object dynamics).

Key insights per tier: Real‑robot data offers the highest alignment and physical fidelity but incurs prohibitive collection cost and limited cross‑embodiment integration. UMI‑style data scales by removing the robot from the capture loop while preserving near‑robotic action supervision, yet still requires calibration and suffers from missing robot‑specific states. Ego‑centric data provides rich human interaction priors at larger scale, trading off annotation precision for coverage. Simulation data is the most scalable and cheap, automatically yielding dense labels, but its physical fidelity is limited, creating a sim‑to‑real gap. Generic data supplies massive semantic, spatial, and reasoning knowledge but lacks robot‑specific motion and contact signals, serving best as a pre‑training foundation.

The paper further discusses "data recipes": how embodied brain models, visual‑language‑action (VLA) models, and world‑action models should select and mix these tiers for pre‑training versus fine‑tuning, how to align heterogeneous action spaces (native robot interfaces, zero‑filled vectors, semantic action slots), and how to reconcile differing coordinate frames (robot base, camera, wrist).

Finally, the authors outline challenges and future directions, emphasizing a shift from merely increasing data volume to improving information coverage: ensuring critical states, contact and failure cases are captured, enabling cross‑embodiment reuse, and sampling data appropriately at each training stage.