Tadpole: Autoencoders as Foundation Models for 3D PDEs with Online Learning

From Predicting the Future to Understanding the World

Foundation models have transformed natural language processing and computer vision. A natural question arises: can the physical world have similar foundation models? Traditional scientific machine learning trains a separate model for each physical system, requiring retraining when the system, scale, or variables change. Foundation models aim to learn universal representations from diverse physical data and transfer quickly to new problems.

Challenge of 3D Physical Data

Three-dimensional simulations produce enormous data. A single high-resolution fluid simulation can occupy massive storage, and training a foundation model demands even larger scale. Different physical systems have varying numbers of variables, spatial resolutions, and evolution laws, making it difficult for one model to learn truly transferable knowledge.

Tadpole: Autoencoder Pretraining Instead of Next-Step Prediction

Many physics foundation models adopt next-step prediction (like language models predicting the next token). Tadpole takes a different route: it is first trained as an autoencoder . Given a 3D physical field at one time step, it compresses the data into a compact latent space and reconstructs the original. Thus, pretraining learns not "what happens next?" but "what do these complex 3D physical states essentially look like?"

This distinction matters. Future evolution depends on specific equations, boundary conditions, and parameters; the same current state can lead to different futures in different systems. Reconstruction focuses on spatial structure, which exhibits strong correlation and continuity across physical systems. The autoencoder learns these universal features first, then transfers them to downstream tasks.

Online Learning: 202 TB of Data Without 202 TB of Storage

Storing and reading massive 3D datasets creates bottlenecks in storage and I/O bandwidth. Tadpole uses online learning : GPU-based numerical solvers generate new physical data in real time during training, feeding it directly to the model instead of pre-saving a static dataset. A multi-level cache mechanism parallelizes data generation, communication, and model training. The pretraining process consumed the equivalent of ~202 TB of training data without ever storing it locally. On high-speed SSDs, online training achieved ~1.8× overall speedup over offline pre-generation; the advantage grows in bandwidth-constrained HPC environments.

Handling Variable Resolutions and Variable Counts via Local Patches

Real-world 3D physical data lacks uniform size: some systems have few variables, others dozens; grid resolutions range from tens to 1024³. Tadpole splits data into local patches. During pretraining, different physical variables are separated, and random 3D crops serve as training samples. This unifies diverse variable counts and spatial sizes into one framework. At inference, the model adjusts patch size to handle resolutions unseen during pretraining.

Experiments show two clear trends: larger models yield better zero-shot reconstruction; more diverse pretraining systems improve generalization to new problems. Tadpole learns transferable 3D physical representations, not just task-specific tricks.

Tadpole-DFT: From Reconstruction to Dynamics Prediction

Since pretraining only learns reconstruction, how does Tadpole predict future evolution? Tadpole-DFT adds minimal new capacity on top of the frozen representation: (1) a lightweight subnet learns inter-variable interactions and temporal dynamics in latent space; (2) skip connections pass fine-grained spatial information directly to the decoder; (3) LoRA fine-tunes only a small fraction of parameters. The goal: preserve pretrained knowledge while incrementally teaching the model "time."

On multiple 3D turbulence prediction tasks, Tadpole-DFT achieves competitive performance. Notably, on turbulent boundary layer tasks, some Tadpole models achieve lower prediction error with far fewer trainable parameters than large baselines. The specialized DFT design utilizes pretrained representations more effectively than full fine-tuning.

Generative Modeling in Latent Space

The learned latent space also enables generation of new 3D physical fields. A generative model trained in the compact latent space produces new latent codes, which Tadpole's decoder reconstructs into full 3D flow fields. This avoids direct generation on huge 3D grids. In 3D turbulence generation experiments, the fine-tuned Tadpole-based generative model achieves strong results on multiple statistical metrics. The same pretrained model thus serves reconstruction, dynamics prediction, and generation — demonstrating true multi-task foundation model capability.

Extension to Unstructured Grids and Complex Geometries

Tadpole's core ideas — reconstruction pretraining and online synthetic data generation — extend beyond regular grids. A follow-up work accepted at NeurIPS 2026, "Geometry Meets Physics: Data-Efficient Pre-Training For Unstructured Neural PDE Solvers" (GMP), applies them to unstructured meshes and complex 3D geometries (e.g., vehicle and aircraft external flows). GMP pretrains from both geometry (learning representations of complex 3D domains for steady problems) and physics (online generation of synthetic transient data). Experiments show faster convergence, higher data efficiency, and better fine-tuning accuracy in data-limited scenarios.

Conclusion

Tadpole explores a different path for physics foundation models: learn a universal physical representation via reconstruction first, generate training data online to bypass storage bottlenecks, then adapt the representation to reconstruction, dynamics prediction, and generation. This raises a broader question: must AI first learn to predict the future to become a foundation model for the physical world? Perhaps another possibility exists: first teach it how to represent the world, then teach it how the world changes.

Paper: https://arxiv.org/abs/2605.15284 (arXiv:2605.15284) Code: https://github.com/tum-pbs/tadpole