Photo‑Level Simulation Bridges Vision Gap for Robot Learning (GS‑Playground, RSS 2026)

GS‑Playground, a new high‑throughput visual‑high‑fidelity simulator jointly developed by Tsinghua University AIR DISCOVER Lab and industry partners, was accepted at RSS 2026, marking a breakthrough in both visual fidelity and training throughput for embodied AI research.

Three core bottlenecks addressed : (1) Rendering cost – existing simulators such as Isaac Lab, ManiSkill, and Genesis exhaust GPU memory when high‑resolution rendering is added, forcing a trade‑off between image quality and scale. (2) Asset creation – building scenes that are both physically and visually accurate still requires extensive manual modeling and engineering. (3) Sim2Real gap – visual and physical discrepancies cause policies trained in simulation to fail on real robots, demanding costly randomization and fine‑tuning.

Architecture redesign consists of three layers:

Self‑developed high‑performance parallel physics engine : based on a speed‑impulse formulation with strict complementarity constraints, supporting both CPU and GPU back‑ends. Compared with PhysX, MuJoCo, and Taichi, it sacrifices gradient smoothness for superior geometric accuracy, enabling stable large‑step simulation (dt=10 ms) and perfect static balance of rigid bodies. Key optimizations include constraint‑island parallelism and time‑coherent hot‑starting, reducing PGS iterations from >50 to <10. In a 50‑object, 27‑DOF humanoid scenario, the engine reaches 1,015 FPS, 32× faster than MuJoCo and ~600× faster than GPU‑based MjWarp.

Efficient batch 3D Gaussian Splatting (3DGS) rendering engine : replaces ray‑tracing or rasterization with 3DGS, adding three modules – a point‑pruning strategy that keeps ~30 % of Gaussian points with <0.05 dB PSNR loss, rigid‑body chain Gaussian kinematics (RLGK) that synchronizes millions of points to low‑dimensional states in sub‑millisecond, and a single‑template broadcast that stores one scene template in GPU memory and broadcasts it to up to 2,048 parallel environments, dramatically lowering memory bandwidth pressure.

Automated Real2Sim asset pipeline : an end‑to‑end “Image‑to‑Physics” workflow that converts a single RGB image into a complete simulation‑ready digital twin. The pipeline chains Grounding‑DINO, SAM1/2, LaMa, AnySplat, SAM‑3D, depth alignment, scale correction, and Speedy‑Splat pruning, producing assets in ~5 minutes per image. Using the Bridge‑v2 dataset, the team released the Bridge‑GS dataset containing 3DGS representations, meshes, 6‑DOF poses, and camera parameters.

Benchmark results : on an RTX 4090 single GPU, rendering 2,048 parallel scenes at 640×480 achieves >10,000 FPS. Across RTX 4090, RTX 6000 Ada, and A100, GS‑Playground consistently outperforms Isaac Sim’s ray‑tracing renderer, which often runs out of memory at higher resolutions.

Sim2Real validation covers four tasks: (1) quadruped walking (Unitree Go2) with 1,024 environments converging in 10 minutes and successful real‑world deployment; (2) humanoid walking (Unitree G1) using 2,048 environments converging in ~6 hours; (3) visual grasping (Airbot Play arm) achieving 90 % zero‑fine‑tuning success on real scenes, whereas MuJoCo, ManiSkill3, and Isaac Lab achieve 0 %; (4) visual navigation (Unitree Go2) using hierarchical RL to navigate directly from first‑person RGB images. The platform also provides the only parallel LiDAR simulation based on 3DGS, supporting RGB, depth, three LiDAR types, and force/contact sensors, with MJCF compatibility.

Significance and outlook : GS‑Playground is not a single‑point improvement but a full‑stack redesign that brings photo‑level visual feedback to a scale previously limited to proprioceptive learning. The team plans to use the platform for large‑scale visual‑language‑action data synthesis and to extend benchmarks for VLA and VLN models. Current limitations include dynamic lighting and soft‑body simulation, which will be addressed by integrating particle‑based dynamics (PBD/MPIM) with Gaussian splatting.