PAT3D: Physics‑Augmented Text‑to‑3D Generates Physically Stable Scenes

Current 3D AIGC systems can quickly generate visually plausible scenes, but they often fail when placed in a physics simulator: objects may float, intersect, or collapse, making the results unsuitable for games, XR, or robotics.

The ICLR 2026 paper PAT3D: Physics‑Augmented Text‑to‑3D Scene Generation (authors: Guying Lin et al., CMU, HKU, HKUST) proposes a three‑stage pipeline to make generated scenes both visually and physically sound.

Stage 1 – 3D Object and Spatial Relation Extraction

The system first creates a reference image from the text prompt, then uses a vision‑language model to identify object categories, materials, and relative positions. The image is segmented into object regions, and a separate 3D asset is generated for each object, allowing every object to act as an independent rigid body in later contact and support calculations.

Stage 2 – Layout Initialization

Using monocular depth estimation, the reference image is back‑projected into a coarse 3D layout. A hierarchical “scene tree” is built from the extracted object dependencies (e.g., support, containment) along the gravity direction. Two corrective passes are applied: horizontal de‑overlap of sibling objects and vertical separation of parent‑child pairs, ensuring an initial layout without interpenetration that is ready for physics simulation.

Stage 3 – Layout Optimization

PAT3D incorporates the differentiable rigid‑body simulator libuipc . Objects evolve under gravity and contact forces toward a static equilibrium. To avoid purely physical solutions that drift from the textual description, a semantic loss is defined based on whether the final simulated state satisfies the “scene tree” relations. This loss is back‑propagated to adjust the initial layout iteratively, yielding scenes that are both stable and semantically faithful.

Experimental Results

On a benchmark of 18 complex prompts, PAT3D was compared with GraphDreamer, Blender‑MCP, and MIDI. It achieved zero displacement after simulation, zero interpenetration, and a physical plausibility score of 88.5, demonstrating that the method delivers fully usable, simulation‑ready environments.

Qualitatively, PAT3D excels in scenes with complex contacts such as books on tables, cups, utensils, building blocks, and fruit baskets. For example, in a block‑stacking scenario, prior methods often produce layouts that collapse under simulation, whereas PAT3D adjusts the initial placement so the final stable configuration still respects the textual description.

Applications

Scene Editing: Users can delete or add objects (e.g., remove a pen holder or insert a new book) and the scene re‑balances in simulation without interpenetration, enabling a “building‑blocks” style of 3D content creation.

Animation Production: Because the generated scenes already satisfy basic physical constraints, they can be directly used for animation without extensive manual layout correction.

Robot Simulation: Physically plausible scenes can be imported into simulators to test grasping and manipulation strategies, providing reliable environments for robot learning and evaluation.

Impact and Availability

PAT3D demonstrates that integrating physics simulation into 3D generation dramatically expands the utility of text‑to‑3D systems beyond static visualizations toward interactive and robotic applications. The source code is released under the Apache‑2.0 license at https://github.com/Simulation-Intelligence/PAT3D, facilitating reproducibility and further research.