Visual Tracks: A New Language for Robot World Models (TrAct)

Li Fei‑Fei and Wu Jiajun’s team propose a new information interface for embodied robot intelligence. Instead of feeding robot actions—low‑dimensional, robot‑specific control signals—into the world model, they place a visual‑track interface between the policy and the predictive model.

Action‑conditioned world models struggle because the same motion can produce very different visual outcomes depending on scene geometry, object states, and contact dynamics. Moreover, existing methods often ignore erroneous actions and rely on visual priors that generate implausible results.

Visual tracks directly describe how task‑relevant points move in image space. The model tracks seven keypoints on the gripper and samples a 5×5 grid of scene points, providing a dense 2‑D trajectory that serves as a clear spatial constraint for future prediction.

TrAct consists of three components:

VLAT (Visual‑Language‑Action Transformer) generates multiple candidate action‑track pairs from the current observation and language instruction, using the π0.5 backbone.

TWM (Trajectory‑Conditioned World Model) predicts future video frames conditioned on the visual tracks. It builds on Stable Video Diffusion and encodes the tracks as spatial control signals via ControlNet.

VLAC (Visual‑Language‑Action Critic) scores each candidate video according to the task instruction, and the system executes the action belonging to the highest‑scoring video. Each decision step samples 20 candidates in simulation and 16 on the real robot.

For comparison, the authors also train an action‑conditioned world model (AWM) that shares the same candidate generation and scoring pipeline; the only difference is the conditioning signal (action vs. visual track).

Quantitative results show the advantage of visual‑track conditioning. On the LIBERO‑INTEGRAL benchmark, the π0.5 baseline achieves 27 % success, VLAT alone reaches 44 %, VLAT + AWM 49 %, while TrAct attains 55 %. In real‑robot experiments with a Franka arm, success rises from 49 % to 76 %. Cross‑robot transfer (Franka → UR5) improves from 17 % to 50 %.

Video quality metrics also improve dramatically: under the simulated external view, PSNR increases from 15.12 to 24.51, LPIPS drops from 0.438 to 0.106, and FVD falls from 129 to 38 when using visual‑track conditioning.

Standard LIBERO tasks are near saturation (≈97 %–98 % success). LIBERO‑INTEGRAL adds object replacement, position changes, task variations, camera viewpoint shifts, and cross‑robot tests, exposing the need for robust generalization. TrAct maintains high success rates under unseen backgrounds and robot bodies, outperforming action‑conditioned models by roughly 20 percentage points on the hardest tasks.

The unified trajectory representation also enables sharing a single prediction target between DROID robot data and EgoDex human first‑person videos. Human hand motions in EgoDex cannot be expressed as robot control commands, so they supervise only the visual track, while robot data supervise both track and action.

Extending pre‑training data further improves performance: on the five most difficult UR5 tasks, TrAct+ raises success from 32 % to 38 %.

In conclusion, TrAct keeps the robot’s final action unchanged but rewires the information flow: visual tracks act as a common language that links control decisions to more accurate future predictions, allowing robots to rehearse before acting.