Roadmap for Physics‑Informed Machine Learning: Lessons from the 2021 Nature Review

Physics‑informed machine learning (PIML)

Pure data‑driven deep learning can fit almost any function but cannot guarantee that predictions obey fundamental physical laws such as mass or energy conservation. Traditional numerical methods (finite element, spectral) enforce physics exactly but struggle with high‑dimensional, multi‑scale, data‑sparse problems. PIML seeks a middle ground where neural networks learn from data while also respecting physics.

Three physical‑embedding pathways

The review classifies physical bias into three standard machine‑learning biases and applies them to PIML:

Observational bias : let the data itself encode physics (e.g., symmetry‑aware data augmentation, multi‑fidelity data fusion, sampling that respects conservation laws).

Inductive bias : hard‑code physical structure into the network architecture (e.g., CNNs for translational invariance, GNNs for multi‑body symmetry, equivariant networks, Hamiltonian networks, hard‑boundary embedding).

Learning bias : add physics‑based terms to the loss function (the classic PINNs approach).

These pathways are not mutually exclusive; the best performance often comes from mixing them, such as using hard‑coded boundary conditions together with soft PDE‑residual penalties and multi‑fidelity observational data.

Analogies between numerical methods and deep learning

Finite‑element basis functions ↔ network activation functions.

Multigrid ↔ ResNet skip connections.

Hierarchical iterative solvers ↔ multi‑level networks / V‑cycle structures.

Implicit Runge‑Kutta ↔ discrete‑time PINNs (up to 500 layers).

Krylov subspace methods ↔ GMRES‑like iterative layers.

ReLU piecewise linear ↔ continuous piecewise‑linear finite elements (proved in He et al., 2020).

Strict equivalences are proven for ReLU ↔ continuous piecewise‑linear FEM and for MgNet ↔ CNN‑multigrid (He & Xu, 2019); other correspondences are heuristic but useful for transferring approximation theory.

From PINNs to neural operators

PINNs learn a solution for a single set of parameters and must be retrained for new parameters. Neural operators (e.g., DeepONet, Fourier Neural Operator) learn the mapping from input functions to output fields, enabling “train once, infer many”.

DeepONet (Lu et al., 2021) uses a branch network for the input function and a trunk network for output coordinates; their inner product yields the predicted field. The review provides a PyTorch implementation:

import torch
import torch.nn as nn

class DeepONet(nn.Module):
    def __init__(self, m=100, branch_layers=[100, 128, 128, 64],
                 trunk_layers=[2, 128, 128, 64]):
        super().__init__()
        b_modules = []
        for i in range(len(branch_layers) - 1):
            b_modules.append(nn.Linear(branch_layers[i], branch_layers[i+1]))
            if i < len(branch_layers) - 2:
                b_modules.append(nn.Tanh())
        self.branch = nn.Sequential(*b_modules)
        t_modules = []
        for i in range(len(trunk_layers) - 1):
            t_modules.append(nn.Linear(trunk_layers[i], trunk_layers[i+1]))
            if i < len(trunk_layers) - 2:
                t_modules.append(nn.Tanh())
        self.trunk = nn.Sequential(*t_modules)

    def forward(self, u_sensors, y_coords):
        """u_sensors: (batch, m) – sensor values of the input function
            y_coords: (batch, 2) – query coordinates (t, x)"""
        branch_out = self.branch(u_sensors)   # (batch, p)
        trunk_out  = self.trunk(y_coords)    # (batch, p)
        return torch.sum(branch_out * trunk_out, dim=-1, keepdim=True)

The Fourier Neural Operator (FNO) is a frequency‑domain global convolution method that excels on regular grids; it receives less coverage because it was only an arXiv preprint when the review was written.

Advantages and limitations of PIML

Scenario 1 – Incomplete models & noisy data : physics acts as regularization, turning ill‑posed inverse problems into well‑posed optimizations.

Scenario 2 – Small‑data strong generalization : embedding physics constrains the solution manifold, enabling extrapolation beyond the training set.

Scenario 3 – Dimensionality‑curse mitigation : neural operators avoid exponential grid growth, though high‑dimensional training remains costly.

Scenario 4 – Mesh‑free handling of irregular domains : PINNs run on TensorFlow/PyTorch without mesh generation.

Well‑posed forward problems : traditional grid‑based solvers still outperform PINNs.

Uncertainty quantification

Standard PINNs provide point estimates only. The review categorizes three uncertainty sources and corresponding methods:

Physical uncertainty – stochastic PDEs; PI‑GANs learn high‑dimensional probability distributions while respecting physics.

Data uncertainty – noisy or missing data; Bayesian PINNs (B‑PINNs) treat network weights as random variables and infer posterior distributions via Hamiltonian Monte Carlo or variational inference.

Model uncertainty – limited network capacity or insufficient training; addressed with ensembles or MC‑Dropout.

Scalability and training challenges

Key failure modes identified include gradient pathology (large scale differences between loss terms) and spectral bias (preference for low‑frequency components). Solutions discussed:

Learning‑rate annealing and dynamic loss weighting (Wang et al., 2021, SIAM JSC).

Neural Tangent Kernel (NTK) analysis explaining why some loss components stagnate (Wang et al., 2022, JCP).

Fourier feature embeddings, multi‑scale architectures, and domain decomposition (XPINNs, Jagtap & Karniadakis, 2020) for large‑scale problems.

Causal PINNs (Wang et al., 2024) to enforce temporal causality in long‑time integration.

Calls for more efficient high‑order automatic differentiation (Taylor‑mode AD).

Software ecosystem

DeepXDE – Python solver (TensorFlow, PyTorch, JAX); supports integer, fractional, integral‑differential equations and complex CSG geometries.

SimNet (now NVIDIA Modulus) – Python solver; GPU‑optimized for large‑scale engineering problems.

NeuralPDE – Julia solver; integrates with the Julia scientific‑computing ecosystem.

SciANN – Python wrapper (TensorFlow); high‑level API, user still implements the solving workflow.

ADCME – Julia wrapper; embeds DNNs into standard numerical schemes (RK, FD, FEM, FV).

GPyTorch / Neural Tangents – Python analysis tools; kernel‑based analysis of PINNs training dynamics.

DeepXDE receives special emphasis because Lu Lu, a core author, contributed heavily to its development.

Practical selection guide

PINNs (MLP) – general purpose, good for limited data and deterministic inference.

PINNs + CNN – suited for 2‑D grid domains.

PINNs + Fourier features – for high‑frequency or periodic PDEs.

PINNs + RNN – for non‑Markovian or temporally discrete problems.

B‑PINNs / probabilistic PINNs – when quantifying epistemic or aleatoric uncertainty.

DeepONet – operator learning; architecture can be MLP, CNN, or RNN depending on data modality.

Sample complexity depends on the strength of inductive bias, compatibility between data and physics, and the intrinsic complexity of the target operator.

Future research directions

Digital twins

Integrating heterogeneous, sparse data streams with physics models to create real‑time digital replicas. Challenges include data heterogeneity, costly preprocessing, and incomplete physics.

Model‑to‑model transformations & explainability

Understanding how different trained models can be mapped onto each other and onto physical representations would improve interpretability.

Intrinsic variable discovery & emergent representations

Beyond PCA, using manifold learning or deep generative models to uncover latent variables that capture essential dynamics, then learning PDEs or ODEs in the emergent space.

Redefining “understanding”

Questioning whether accurate predictions without mechanistic insight constitute scientific understanding, especially as large language models become more prevalent in scientific workflows.

Impact and position

The review is described as the “map” that followed the 2019 PINNs paper’s “match”. Its three‑bias framework has become standard terminology in PIML literature. Citation counts have risen dramatically (96 in 2021 → >2,800 in 2025), indicating rapid growth. The authors acknowledge a self‑citation bias: PINNs and DeepONet receive more coverage than competing methods such as FNO, SINDy, or Deep Ritz, so readers should consult complementary surveys for a balanced view.

Limitations

While the roadmap is comprehensive, it under‑represents alternative data‑driven discovery approaches (e.g., SINDy) and recent theoretical advances in deep‑learning‑based numerical methods. Benchmarking efforts like PINNBench and PDEBench have begun to address the authors’ call for standardized tests, but a universally accepted benchmark is still lacking.