Zero-HLM: 45 Demos to 4,800 Valid Trajectories, 69% Zero-Shot Humanoid Success
Zero-HLM, a Real2Sim2Real framework from BIGAI, BIT, Tsinghua, and Deta, reconstructs real scenes via Gaussian splatting and GPT-Astra, then generates and validates whole-body humanoid demonstrations in simulation, expanding 45 source demos into 4,800 executable trajectories that enable zero-shot transfer to a Unitree G1 with 69% success across five generalization settings, outperforming real-data baselines (23%).
Overview
Zero-HLM is a Real2Sim2Real pipeline developed by the Beijing Institute for General Artificial Intelligence (BIGAI), Beijing Institute of Technology, Tsinghua University, and Deta Intelligence. It addresses two core barriers to sim-to-real whole-body humanoid manipulation: (1) building a physically interactive simulation environment that matches visual appearance, and (2) generating executable whole-body demonstrations that survive execution verification.
Scene Reconstruction: From Phone Video to Interactive Simulation
The process starts with multi-view phone footage of a real room. Gaussian splatting creates a photorealistic 3D representation that can be viewed from arbitrary angles. However, visual fidelity alone is insufficient; the robot needs collidable geometry, articulated joints (e.g., cabinet doors), and physical properties. GPT-Astra iteratively aligns the simulation's shapes, poses, collision meshes, and joint configurations to the Gaussian reconstruction, ensuring that the table the robot sees coincides with the surface it contacts.
Demonstration Generation with Execution Feedback
After scene construction, operators teleoperate the humanoid in simulation to collect 15 successful demonstrations per task (45 total across three tasks). For each new configuration (different start poses, object/goal positions), Zero-HLM adapts the reference motions and re-plans the mobile base trajectory. Crucially, every generated demonstration is executed by the same whole-body controller (SONIC) that will be used on the real robot. The execution is checked for balance, collisions, and task completion.
If execution fails, the system uses the feedback to co-adjust both the simulation scene (e.g., shifting object positions to restore required spatial relationships) and the reference motion (e.g., compensating wrist height). The adjusted demonstration is re-executed and only added to the dataset upon success. This automated loop expands the 45 source demonstrations into 4,800 verified successful trajectories without human intervention.
Real-World Experiments on Unitree G1
The generated dataset trains a GR00T N1.7 policy (pre-trained model) deployed via the SONIC whole-body controller on a Unitree G1 humanoid. Three long-horizon tasks are evaluated:
Task A: Walk to table, place banana in basket.
Task B: Squat, pick up hat, place on cabinet.
Task C: Two-hand carry fruit basket, close door with foot, place basket on stool.
Evaluations cover base configurations (matching the original demonstration distribution) and extended configurations (varying start poses, grasp/place positions, object-goal combinations). Each configuration is tested 20 times per task (100 total trials for extended settings).
Results
Base config (100 demos each): Simulated data (43.3%) matches real-data baseline (46.7%).
Base config (800 demos each): Simulated data reaches 66.7% average success — 20 percentage points higher than the 100 real-demo baseline. Per-task: 60%, 75%, 65%.
GPT-Astra baseline: 20% on Task A, 0% on Task B, no usable demos for Task C, highlighting the difficulty of generating stable whole-body demos without execution feedback.
Extended config (5 settings, 100 trials total): Real-data baseline succeeds 23 times (23%); Zero-HLM succeeds 69 times (69%). Breakdown: start-pose change → 71.4% (banana), 70% (hat); grasp/place change → 67.5%, 50%; object-goal recombination → 100% (5/5) vs. 20% (1/5) for real baseline.
Zero-HLM-Bench: Unified Benchmark
The team open-sources Zero-HLM-Bench, which decomposes the three long-horizon tasks into nine skill stages. A unified interface enables comparison of policies (π₀.₅, GR00T N1.7, DiT4DiT) and pinpoints failure stages (navigation, grasp, transport, place). Under the same protocol, GR00T N1.7 achieves 69.7% average full-task success, DiT4DiT 64.0% (80% on hat transport), π₀.₅ 59.0%.
Limitations and Future Work
Diagnostic experiments reveal that while simulation action supervision transfers well, the visual domain gap remains a major bottleneck. Naively mixing real and simulated data does not automatically yield optimal performance; aligning visual distributions and coordinating training across domains requires further research. Current validation covers three tasks and their configuration variations; future work must test more objects, longer horizons, environmental changes, and external disturbances for household and factory deployment.
Resources
Paper: https://arxiv.org/abs/2610.35075
Project page: https://young-bit.github.io/zero-hlm/
Code: https://github.com/BIGAI-General-Vision-Lab/Zero-HLM
Signed-in readers can open the original source through BestHub's protected redirect.
This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactand we will review it promptly.
How this landed with the community
Was this worth your time?
0 Comments
Thoughtful readers leave field notes, pushback, and hard-won operational detail here.
