Why Robots Shouldn’t Dream in Pixels: Introducing μ₀’s 3D Interaction Traces as a Physical Language

Embodied AI and world models have become hot topics, leading many to wonder whether robots should learn a world model by training ever larger video‑prediction models that “dream” in pixel space.

While pixel‑level models are directly scalable because video data is abundant, they focus on low‑level details (lighting, texture, camera motion) that are irrelevant to a robot’s core questions such as object motion, contact events, and tool‑object relationships.

This raises a chicken‑and‑egg problem: training a robotics‑specific pixel world model requires massive robotics data, yet that same data could be used to train a policy directly, undermining the purpose of a world model.

Latent world models avoid predicting every pixel by compressing the scene into a compact latent space, but they often become black‑boxes that are hard to interpret, intervene with, or correct when they fail.

The authors of μ₀ therefore ask: is there a representation that is neither as low‑level and redundant as pixels nor as opaque as latent vectors? Their answer is “3D interaction traces”.

μ₀ is a symbolic/structured world model that predicts the motion of a small set of semantic interaction points—object parts, tools, hands, and contact regions—collectively called 3D interaction traces. Each trace corresponds to a meaningful physical entity (e.g., an object edge or a fingertip contact area), making the representation compact yet interpretable.

The authors view these traces as a “physical language”: a set of symbols that describe how objects move during interaction, analogous to how words form a shared language for large language models.

To learn this representation, μ₀ introduces a data engine called TraceExtract. The pipeline first detects “what is moving”, then estimates “where it moves”, and finally decomposes “how it moves”, converting ordinary video into trace supervision without requiring expensive robot action labels.

For academic labs lacking industrial‑scale compute and data, μ₀’s pre‑training uses about 200 K episodes, 13 M frames, and 15.7 TB of video—substantial for a university cluster but far smaller than the datasets used for industrial VLA models.

The training strategy keeps a vision‑language backbone for semantic knowledge while a separate trace expert learns physical dynamics. Crucially, the pre‑training phase does not need action labels; after freezing μ₀, a lightweight action expert maps trace features to robot commands.

Experiments show that μ₀ excels at trace forecasting across multiple horizons and metrics, running inference in roughly 0.29 seconds per prediction. When combined with a frozen trace model and a lightweight action expert, it achieves robot performance comparable to strong VLA policies, with an average success rate surpassing π₀.₅ in real‑world evaluations.

The key insight is that the value of a world model for robotics may lie not in generating realistic videos but in learning a transferable, interpretable, and intervene‑able physical representation.

3D interaction traces are only a first step; future work could incorporate contact graphs, force/torque traces, tactile fields, object‑centric affordance graphs, constraints, and energy landscapes—representations that are less “universal” than pixels yet closer to the physical language robots need.

Before scaling data and model size, the authors argue that choosing the right symbol space is essential; scaling the wrong representation would merely waste resources.

In summary, μ₀ demonstrates that robots should move beyond pixel‑level dreaming and black‑box latents toward their own symbolic space—traces—that can serve as a physical language for embodied intelligence.