How an Agentic Loop Turns Text‑to‑3D Scene Generation into an Iterative Planning Process

Large language models are entering an "Agent era" where they must not only speak or write but also plan and execute actions. For embodied intelligence, the high cost of real‑world trial‑and‑error makes a physically plausible, structurally sound 3D world essential.

Generating a usable 3D scene from a textual description is far harder than producing a few appealing images. A scene must contain correctly placed objects—cups on tables, books on shelves, chairs oriented sensibly—and avoid interpenetration, floating, or unstable support. The difficulty lies in ensuring spatial relationships that are both realistic and functional for interaction and simulation.

Previous work follows two main routes. The first relies on datasets such as 3D‑FRONT to train models that can arrange indoor layouts, but these models are locked to the training distribution and struggle to generalise to outdoor environments or fine‑grained relations like "small object inside a large object." The second uses large language models for open‑ended layout planning; while they capture semantic intent, they operate only in symbolic space, often producing layouts where chairs face walls, cabinets block windows, or objects float when instantiated in geometry.

Scenethesis proposes a new hybrid approach that closes the loop between language, vision, and physics. The system consists of four stages:

Stage 1 – Semantic Planning: A language model parses the text prompt, identifies the scene type, selects key anchor objects, and builds an initial hierarchical layout. The output is a JSON list of chosen objects together with an expanded scene description.

Stage 2 – Visual Grounding: The visual module generates reference images, performs instance segmentation and depth estimation, and recovers the initial 3D size of each object. This converts the abstract semantic layout into concrete spatial cues grounded in real‑world visual statistics.

Stage 3 – Physical Optimization: Using a signed distance field (SDF), the system refines geometry to satisfy contact, support, and stability constraints. This fine‑grained alignment eliminates floating, interpenetration, and unstable configurations, ensuring that small objects are truly placed inside larger ones rather than merely appearing close.

Stage 4 – Self‑Check and Repair: A judge module evaluates object categories, spatial relations, and overall consistency. If the scene fails the check, the system loops back to re‑plan and repair. This generate‑check‑repair cycle raises the first‑round success rate from about 72 % to 91 % after the self‑check.

Experimental results show that the closed‑loop design not only makes scenes look more realistic but also dramatically improves physical plausibility: collision rates drop from 6.1 % to 0.8 %, and the system can handle richer spatial relations such as "object on top of" or "inside" across indoor and outdoor domains (beach, street, park). This capability is valuable for virtual content creation, simulation environment construction, and training embodied agents that require manipulable, editable worlds.

Limitations remain, including dependence on the diversity of the asset library, reduced precision under heavy occlusion, and limited support for dynamic structures. Nevertheless, Scenethesis demonstrates a promising direction: moving from a single‑shot generation to a multimodal, physics‑aware, iterative workflow that brings us closer to truly interactive 3D world generation.