ULTRA: Unified Control for Humanoid Tracking & Goal-Driven Manipulation

Introduction

Moving a box from the floor to a target location is trivial for humans but requires full-body coordination for humanoid robots. The robot must adjust stance, maintain balance while squatting, grasp the box securely, and coordinate legs, torso, and arms during lifting and transport. Any error can cause the box to slip or the robot to fall.

Problem: Tracking vs. Goal-Driven Control

One approach is to provide a full reference trajectory for the robot to track frame by frame. However, when only a sparse goal (e.g., target object position and orientation) is given, the intermediate motions must be generated autonomously. The core question: can a single controller both track reference motions precisely when they are available and generate motions to achieve sparse goals when they are not?

ULTRA: Unified Multimodal Control

ULTRA (Unified Multimodal Control for Autonomous Humanoid Whole-Body Loco-Manipulation) addresses this by unifying motion tracking and sparse goal control within a single policy. The same policy parameters handle both modalities and can accept different perception inputs, including external motion-capture object states and the robot's own depth camera observations. The work was accepted at IROS 2026 and nominated for the Mobile Manipulation Paper Award.

Physics-Driven Neural Motion Retargeting

Human motion capture data provides rich whole-body demonstrations, but direct mapping to a robot fails due to differences in proportions, joint structure, and contact dynamics. ULTRA introduces physics-driven neural motion retargeting: a reinforcement learning policy that learns to reproduce the demonstrated motion in simulation while respecting dynamics and contact constraints. The retargeting policy preserves the human motion intent and generates robot-feasible trajectories. After training, the same policy can process new motions and be augmented by varying motion ranges and object sizes without retraining.

Two-Stage Policy Learning

Stage 1: Teacher-Student Policy Distillation

A teacher policy is trained with full simulation state and reference motions to learn coordination, contact maintenance, and task completion. These capabilities are then distilled into a student policy that must operate with varying input combinations: sometimes full reference, sometimes only sparse goals; sometimes precise object state, sometimes only first-person depth perception. An input availability mask informs the student which inputs are present, enabling a single model to adapt to different sensing conditions.

Stage 2: Reinforcement Learning Fine-Tuning

After imitation, the policy is further fine-tuned with RL in simulation to handle deviations: initial state variations, new goal positions, and execution errors. The training objective shifts from reproducing reference motions to achieving the actual task goal, improving robustness to distribution shift.

First-Person Depth Perception for Autonomous Operation

Beyond control, ULTRA addresses perception: instead of relying on external motion capture, the robot uses its onboard depth camera to extract object point clouds, which are fed to the control policy along with proprioception and task goals. Experiments on the Unitree G1 humanoid evaluated full motion tracking, fine-grained control, long-horizon goal following, and goal-driven control with first-person point clouds. The results show that RL fine-tuning significantly increases success rates when using first-person point cloud input.

Conclusion

ULTRA demonstrates a viable path to connect human motion data, control, and perception, enabling a single policy to accept task specifications of varying detail and act on diverse sensory inputs. By letting the controller fill in the "how" when only the "what" is specified, ULTRA advances autonomous humanoid loco-manipulation.

References

Paper: ULTRA: Unified Multimodal Control for Autonomous Humanoid Whole-Body Loco-Manipulation (arXiv:2603.03279) - https://arxiv.org/abs/2603.03279

Project page: https://ultra-humanoid.github.io/