How JD Explore’s Silver‑Bullet‑3D Dominated the SAPIEN ManiSkill Challenge

JD Explore Research Institute’s Visual and Multimedia Lab team “Silver‑Bullet‑3D” achieved outstanding results in the 2021 SAPIEN ManiSkill Challenge, ranking first in the No Interaction Track and second in the No Restriction Track, demonstrating the institute’s strong capabilities in robotic‑arm perception and control.

The SAPIEN ManiSkill Challenge, co‑organized by the University of California‑San Diego, UC Berkeley and Stanford University and presented at ICLR 2022, featured four tasks—opening cabinet doors, moving chairs, opening drawers, and transporting buckets—and attracted more than 30 teams from leading institutions worldwide.

In the No Interaction Track, which requires imitation‑learning solutions, the team employed a two‑module approach: behavior cloning of demonstrated actions and offline reinforcement learning, combined with a Transformer‑based relational modeling network to capture complex textures and structures of the robot and objects. This model ranked first across all four tasks in both the first and second evaluation phases.

For the No Restriction Track, allowing rule‑based solutions with full interaction in the simulation, the team proposed a heuristic‑rule method (HRM) that decomposes complex tasks into sub‑tasks, uses observed images and point clouds, and predicts robot actions based on rule‑based control logic. Their solution won first place with a 23.8% absolute margin over the runner‑up.

JD Explore Research Institute holds 18 patents in computer‑vision technologies, has published over 180 papers at top AI conferences, and earned best demo and best open‑source project awards at ACM Multimedia for multimodal interaction and cross‑modal analysis.

The team’s success highlights the rapid advancement of mechanical‑arm perception and control, suggesting that future intelligent‑arm platforms built on 3D vision and robotics will significantly boost logistics, industrial IoT, food safety monitoring, and broader digital‑intelligence applications.