Skip to content
Zero-HLM: 45 Demos to 4,800 Valid Trajectories, 69% Zero-Shot Humanoid Success

Zero-HLM: 45 Demos to 4,800 Valid Trajectories, 69% Zero-Shot Humanoid Success

Zero-HLM, a Real2Sim2Real framework from BIGAI, BIT, Tsinghua, and Deta, reconstructs real scenes via Gaussian splatting and GPT-Astra, then generates and validates whole-body humanoid demonstrations in simulation, expanding 45 source demos into 4,800 executable trajectories that enable zero-shot transfer to a Unitree G1 with 69% success across five generalization settings, outperforming real-data baselines (23%).

Machine Heart
Machine Heart
Machine Heart

Overview

Zero-HLM is a Real2Sim2Real pipeline developed by the Beijing Institute for General Artificial Intelligence (BIGAI), Beijing Institute of Technology, Tsinghua University, and Deta Intelligence. It addresses two core barriers to sim-to-real whole-body humanoid manipulation: (1) building a physically interactive simulation environment that matches visual appearance, and (2) generating executable whole-body demonstrations that survive execution verification.

Scene Reconstruction: From Phone Video to Interactive Simulation

The process starts with multi-view phone footage of a real room. Gaussian splatting creates a photorealistic 3D representation that can be viewed from arbitrary angles. However, visual fidelity alone is insufficient; the robot needs collidable geometry, articulated joints (e.g., cabinet doors), and physical properties. GPT-Astra iteratively aligns the simulation's shapes, poses, collision meshes, and joint configurations to the Gaussian reconstruction, ensuring that the table the robot sees coincides with the surface it contacts.

Figure 1: Zero-HLM overall framework: scene reconstruction, demonstration generation, execution verification, and real-world deployment.
Figure 1: Zero-HLM overall framework: scene reconstruction, demonstration generation, execution verification, and real-world deployment.

Demonstration Generation with Execution Feedback

After scene construction, operators teleoperate the humanoid in simulation to collect 15 successful demonstrations per task (45 total across three tasks). For each new configuration (different start poses, object/goal positions), Zero-HLM adapts the reference motions and re-plans the mobile base trajectory. Crucially, every generated demonstration is executed by the same whole-body controller (SONIC) that will be used on the real robot. The execution is checked for balance, collisions, and task completion.

If execution fails, the system uses the feedback to co-adjust both the simulation scene (e.g., shifting object positions to restore required spatial relationships) and the reference motion (e.g., compensating wrist height). The adjusted demonstration is re-executed and only added to the dataset upon success. This automated loop expands the 45 source demonstrations into 4,800 verified successful trajectories without human intervention.

Figure 2: Three real-robot tasks and their base/extended configurations.
Figure 2: Three real-robot tasks and their base/extended configurations.

Real-World Experiments on Unitree G1

The generated dataset trains a GR00T N1.7 policy (pre-trained model) deployed via the SONIC whole-body controller on a Unitree G1 humanoid. Three long-horizon tasks are evaluated:

Task A: Walk to table, place banana in basket.

Task B: Squat, pick up hat, place on cabinet.

Task C: Two-hand carry fruit basket, close door with foot, place basket on stool.

Evaluations cover base configurations (matching the original demonstration distribution) and extended configurations (varying start poses, grasp/place positions, object-goal combinations). Each configuration is tested 20 times per task (100 total trials for extended settings).

Results

Base config (100 demos each): Simulated data (43.3%) matches real-data baseline (46.7%).

Base config (800 demos each): Simulated data reaches 66.7% average success — 20 percentage points higher than the 100 real-demo baseline. Per-task: 60%, 75%, 65%.

GPT-Astra baseline: 20% on Task A, 0% on Task B, no usable demos for Task C, highlighting the difficulty of generating stable whole-body demos without execution feedback.

Extended config (5 settings, 100 trials total): Real-data baseline succeeds 23 times (23%); Zero-HLM succeeds 69 times (69%). Breakdown: start-pose change → 71.4% (banana), 70% (hat); grasp/place change → 67.5%, 50%; object-goal recombination → 100% (5/5) vs. 20% (1/5) for real baseline.

Figure 3: Task success rates on real G1.
Figure 3: Task success rates on real G1.

Zero-HLM-Bench: Unified Benchmark

The team open-sources Zero-HLM-Bench, which decomposes the three long-horizon tasks into nine skill stages. A unified interface enables comparison of policies (π₀.₅, GR00T N1.7, DiT4DiT) and pinpoints failure stages (navigation, grasp, transport, place). Under the same protocol, GR00T N1.7 achieves 69.7% average full-task success, DiT4DiT 64.0% (80% on hat transport), π₀.₅ 59.0%.

Figure 4: Skill-stage evaluation under varying start poses and object positions.
Figure 4: Skill-stage evaluation under varying start poses and object positions.
Figure 5: Full-task success rates of different models in Zero-HLM-Bench (simulation).
Figure 5: Full-task success rates of different models in Zero-HLM-Bench (simulation).

Limitations and Future Work

Diagnostic experiments reveal that while simulation action supervision transfers well, the visual domain gap remains a major bottleneck. Naively mixing real and simulated data does not automatically yield optimal performance; aligning visual distributions and coordinating training across domains requires further research. Current validation covers three tasks and their configuration variations; future work must test more objects, longer horizons, environmental changes, and external disturbances for household and factory deployment.

Resources

Paper: https://arxiv.org/abs/2610.35075

Project page: https://young-bit.github.io/zero-hlm/

Code: https://github.com/BIGAI-General-Vision-Lab/Zero-HLM

Original Source

Signed-in readers can open the original source through BestHub's protected redirect.

Sign in to view source
Republication Notice

This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactadmin@besthub.devand we will review it promptly.

humanoid robotGaussian SplattingSim-to-Real TransferReal2Sim2RealWhole-Body ManipulationGPT-AstraZero-HLMZero-HLM-Bench
Reader feedback

How this landed with the community

Sign in to like

Rate this article

Was this worth your time?

Sign in to rate
Discussion

0 Comments

Thoughtful readers leave field notes, pushback, and hard-won operational detail here.