PhysLDM: Latent Diffusion Paradigm for High-Fidelity Deformable Simulation

Researchers from Nanyang Technological University MMLab and Shanghai AI Laboratory present PhysLDM , a latent diffusion paradigm for high-fidelity deformable simulation. The paper ( https://arxiv.org/abs/2610.07609) and project page ( https://yucrazing.github.io/physldm/) demonstrate that a single model can generate entire high-resolution simulation trajectories with millimeter-level reconstruction error, zero-shot generalization to unseen objects, and end-to-end differentiability.

Why Physics Simulation Remains a Hard Problem for Neural Networks

The authors identify four interlocking challenges that have prevented neural simulators from replacing traditional solvers in games, animation, and robotics:

Frame-by-frame drift: Autoregressive models (e.g., MeshGraphNets) accumulate error per step, causing long-term trajectory divergence.

Scale explosion: High-fidelity volumetric meshes contain thousands to tens of thousands of vertices; a Transformer attending over dozens of frames hits quadratic attention complexity. The team measured OOM at 2,600 vertices on a 140 GB H200 without compression.

No ready-made spatio-temporal compression: Prior work compressed single frames; video VAEs transferred poorly to physics trajectories.

Poor generalization: Neural simulators famously fail when the test object differs from training objects.

These issues are coupled: eliminating drift requires one-shot prediction, which demands compression, which must be both accurate and generalizable.

Building a Spatio-Temporal VAE for Physics

Analogous to Stable Diffusion's image VAE, PhysLDM first learns a tokenizer that compresses an entire simulation trajectory into a compact latent representation. The team iterated through several failed designs:

Naïve temporal striding (borrowed from video VAEs) produced "staircase" artifacts — the deformation would pause then jump.

Adding velocity loss, spectral loss, or adaptive temporal queries did not remove the artifacts.

Compressing world coordinates directly scrambled geometric detail; sharing temporal compression between rigid motion and deformation amplified translation error and flattened deformation.

The final design incorporates two key ideas:

Rigid-deform decoupling: Each frame is split into a global rigid transformation plus local deformation. The network only learns the millimeter-scale deformation.

Global temporal compression: A set of learnable temporal queries aggregates the entire trajectory into a fixed number of tokens, regardless of vertex count, achieving up to 78× compression .

Training data: ~80,000 trajectories generated on 30,000+ Objaverse objects using the GPU FEM solver StiffGIPC . The spatio-temporal VAE achieves an average reconstruction error of 2.48 mm .

Zero-Shot Generalization Across Objects, Resolutions, and Physical Models

After training on Objaverse, PhysLDM is evaluated zero-shot on multiple out-of-distribution settings:

Unseen object datasets (GSO, Toys4K): reconstruction error stays at training-distribution level.

High-resolution meshes exceeding twice the training vertex limit: error ~ 5 mm .

Different energy model (ARAP): error 4.6 mm .

Real captured point-cloud trajectories : error 6 mm .

The authors attribute this to the massive, diverse training geometry and the purely kinematic supervision that does not tie the model to any specific constitutive model.

Diffusion Outperforms Regression in Chaotic Deformable Dynamics

A counter-intuitive finding motivates the generative approach: even traditional solvers are non-deterministic under chaos. Running the same byte-identical scene five times on the GPU solver produced final states differing by up to 442 mm due to microscopic floating-point differences amplified by chaotic deformation.

When multiple valid futures exist (e.g., a falling key tipping left or right), a regression model averages them into a physically implausible "mean" trajectory. PhysLDM's diffusion model samples distinct valid modes. Quantitatively, PhysLDM's physical consistency significantly outperforms regression and autoregressive baselines; the autoregressive error is nearly 3× that of PhysLDM.

End-to-End Differentiability Enables Inverse Design and Material Inference

Because the entire pipeline is differentiable and supports higher-order gradients, PhysLDM can solve inverse problems rapidly:

Material stiffness estimation from a single trajectory in ~40 seconds — existing differentiable simulators require 20+ minutes or fail.

Second-order optimization (e.g., "how to throw an object to best reveal its material") — comparable tasks take days with current differentiable simulators.

The real point-cloud zero-shot reconstruction also hints at learning physics directly from real-world data.

Multi-Object Extension and Community Reception

Without architectural changes, PhysLDM extends to multi-object scenes. Early feedback from robotics practitioners highlights the potential for automating soft-body physics tuning, and researchers note that diffusion is undergoing a "modality explosion" with PhysLDM adding "deformable mesh trajectories" as a new modality.

Conclusion

PhysLDM brings high-fidelity physics simulation onto the generative AI roadmap: a spatio-temporal VAE compresses whole trajectories, a diffusion model generates them in one shot, achieving millimeter accuracy with zero-shot generalization. The work reveals that in chaotic deformable dynamics, diffusion captures the multi-modal nature of physics better than regression, and its end-to-end differentiability opens doors for inverse design, material inference, and learning from real data — a significant step toward a physical world model.

}