HSImul3R: Physics-in-the-Loop Reconstruction Turns Human Videos into Robot Skills

Problem: Visual Correctness ≠ Physical Validity

Traditional 3D reconstruction focuses on geometric accuracy and visual alignment, but a visually plausible human sitting on a chair may collapse in physics simulation due to missing structural support or interpenetration. The authors state: "Visual correctness does not imply physical interaction correctness."

HSImul3R: Physics-in-the-Loop Bidirectional Optimization

HSImul3R, accepted at ECCV 2026, introduces a Physics-in-the-Loop framework that makes the physics simulator an active supervisor during reconstruction. It operates in two directions:

Forward (Scene-targeted Reinforcement Learning): Optimizes human motion to be both physically feasible and stably interacting with the current scene, preserving intended contacts (e.g., actually sitting on the specific chair).

Backward (Direct Simulation Reward Optimization, DSRO): Uses simulation feedback to refine the 3D scene geometry, evaluating objects not just on standalone stability but on whether they support stable human-object interaction.

HSIBench: Benchmarking Interaction Stability

The authors constructed HSIBench , comprising 19 scene objects, over 50 human motion sequences, and 300 interaction instances captured from 16 synchronized views by 3 participants. The core metric is Stability-HSI , requiring the object to remain stable under gravity, maintain stability throughout the interaction, and preserve meaningful human-object contact.

Quantitative Results

On Easy, Medium, and Hard tasks, HSImul3R achieves interaction stability rates of 53.68% , 30.56% , and 13.92% , respectively, compared to HSfM's 10.52% , 4.50% , and 2.66% . Human-scene 3D penetration rate drops from 69.51% to 22.90% .

Sim-to-Real Deployment on Unitree G1

Optimized human motions are retargeted to the Unitree G1 humanoid robot. A whole-body control policy is trained in simulation using these motions as priors and deployed directly on the real robot, demonstrating the full pipeline: image/video → 3D human-scene reconstruction → physics optimization → robot motion transfer → real-world execution .

Implications

This work enables large-scale human video datasets to become sources of simulation-ready, learnable, executable robot skill assets , moving beyond visual imitation to understanding why an action succeeds in the physical world.

Paper: HSImul3R: Physics-in-the-Loop Reconstruction of Simulation-Ready Human–Scene Interactions (ECCV 2026). ArXiv: https://arxiv.org/abs/2603.15612. Project: https://yukangcao.github.io/HSImul3R/. GitHub: https://github.com/yukangcao/HSImul3R.