Composing 3D objects into realistic scenes is important for digital content creation and robot simulation, but current methods struggle with tasks that require physical interaction, such as hanging, settling under gravity, or hidden containment. Existing approaches typically rely on large language models to predict coordinates or on diffusion models to generate visual alignments, both of which lack a grounded understanding of physics and interactive construction.

PhySGA addresses this by embedding physical simulation into an agentic loop. An Assembly Agent handles geometric and kinematic reasoning for step planning and coarse adjustment, while a Critic Agent reasons over contact dynamics to diagnose simulator states and make fine-grained corrections. Action Routers translate the agents' semantic plans into executable physical primitives inside the simulator.

The framework also includes a Self-Evolving Skill Library that distills past trial-and-error experience into reusable strategies, reducing the number of exploration attempts needed. In experiments conducted in Isaac Sim, PhySGA significantly outperformed existing methods across relation fidelity, physical plausibility, and visual quality, demonstrating the value of grounding composition in interactive physics rather than static priors.