How Knowledge Distillation Lets Neural Networks Grow Physical Symmetry Without Hard PINN Constraints

Soft constraints in PINNs

Physics‑informed neural networks (PINNs) embed PDE residuals in the loss function. The residual term drives the average residual toward zero, but the network may still violate exact symmetries (e.g., an antisymmetric Burgers solution) because the constraint is only soft.

Standard fully‑connected PINNs use the same topology for symmetric Laplace problems and for arbitrary random functions; physical information resides solely in the loss.

Gradient conflict with regularization

Adding parameter regularization (L1, L2, GrOWL) to the PINN loss increases the total loss and degrades accuracy. The PDE residual gradient (∇L_PDE) pushes parameters toward satisfying the differential equation, while the regularization gradient (∇Ω(θ)) pushes parameters toward zero or sparsity; the two directions can be antagonistic.

Figure: Adding regularization to a standard PINN creates a severe gradient clash between the PDE term and the regularization term.

为什么正则化在 PINNs 中失效？
  标准 PINN 的损失:  L = L_data + λ·L_PDE
  加正则化后的损失:  L = L_data + λ·L_PDE + β·Ω(θ)

  问题:
  ┌─────────────────────────────────────────────────┐
  │ L_PDE 的梯度方向: 让网络预测满足物理方程 │
  │ Ω(θ) 的梯度方向: 让网络参数趋向于零/稀疏 │
  └─────────────────────────────────────────────────┘

Knowledge‑distillation decoupling

The authors separate the two objectives into two networks:

Teacher network : a standard PINN trained with data and PDE residuals; provides physics‑consistent predictions.

Student network : learns the teacher’s output via a distillation loss and is regularized with L2; contains no PDE residual term, so the gradient conflict disappears.

Figure: Ψ‑NN separates physical constraints (teacher) from parameter regularization (student), removing the gradient clash.

Ψ‑NN three‑step framework

Distillation : the teacher PINN predicts the full field; the student mimics these predictions while L2 regularization encourages weight clustering.

Structure extraction : after training, the student’s weight matrix exhibits clusters. Hierarchical Agglomerative Clustering (HAC) on the absolute values of the weights yields cluster centroids that encode shared, sign‑flipped, or permuted connections.

Network reconstruction : the extracted relationship matrix R is embedded into a new network; only the independent parameters (non‑zero entries of R) are re‑initialized, dramatically reducing the total parameter count.

Theoretical guarantees

Theorem 1 : In a hidden layer, trainable parameters that play equivalent roles converge to the same value under L2 regularization. Theorem 2 : In a hidden layer, parameters that are symmetric (identical magnitude, opposite sign) converge to equal absolute values under regularization.

These results ensure that physical symmetries leave detectable fingerprints in the parameter space.

Structure extraction details

Extraction proceeds as follows:

Take absolute values of the weight matrix (to capture sign‑agnostic symmetry).

Apply HAC, which automatically determines the number of clusters via a dendrogram.

Replace each cluster’s weights by its centroid while preserving original signs.

Construct a relationship matrix R that encodes three possible relations: shared rows, sign‑flipped rows, or row permutations.

For the 2‑D Laplace problem, the second hidden‑layer matrix reveals an “anti‑diagonal sharing” pattern that emerges automatically from data and physics.

Network reconstruction

Using R, a new network is built where R is frozen (non‑trainable) and only the independent parameters are re‑initialized. This yields a dramatically smaller model because many weights are tied by sharing, sign‑flipping, or permutation constraints.

Laplace 方程的 Ψ‑NN 重构网络
  输入: (x₁, x₂)
       │
       ▼
  ┌──────────────────────────────┐
  │ 第一层:                       │
  │  W₁ = [W₁ᵃ  W₁ᵇ]              │
  │       [W₁ᵃ  W₁ᵇ]   ← 行复制   │
  │  b₁ = [b₁ᵃ, b₁ᵃ]ᵀ            │
  └──────────┬───────────────────┘
           │
           ▼
  ┌──────────────────────────────┐
  │ 第二层:                       │
  │  W₂ = [W₂ᵃ  W₂ᵇ]              │
  │       [W₂ᵇ  W₂ᵃ]   ← 反对角交换│
  │  b₂ = [b₂ᵃ, -b₂ᵃ]ᵀ           │
  └──────────┬───────────────────┘
           │
           ▼
  ┌──────────────────────────────┐
  │ 输出层:                       │
  │  W₃ = [W₃ᵃ  W₃ᵃ]   ← 参数共享│
  └──────────┬───────────────────┘
           │
           ▼
  输出: u(x₁, x₂)

The reconstructed architecture encodes the spatial antisymmetry of the Laplace solution without any explicit loss‑function constraint.

Comparison with hand‑engineered symmetry methods

Prior knowledge requirement : hand‑engineered methods need a known symmetry type; Ψ‑NN only needs the PDE form.

Structure design : manual construction vs. automatic discovery from data and physics.

Applicable range : specific to a problem vs. transferable to other problems with the same PDE family.

Interpretability : depends on designer’s insight vs. directly visible in the relationship matrix R.

Experimental validation

Four benchmark problems were evaluated on an Intel 12400F CPU + RTX 4080 GPU using the Adam optimizer.

Laplace equation (symmetry emergence)

Full‑field L2 error: PINN = 1.159 × 10⁻³ (baseline); PINN‑post = 4.017 × 10⁻⁴ (≈65 % improvement); Ψ‑NN = 0.7422 × 10⁻⁴ (≈93.6 % improvement).

Ψ‑NN converged in roughly half the iterations required by PINN and required ≈1.5 × 10⁴ fewer steps than PINN‑post.

Boundary error remained low because the symmetry is built into the architecture.

Burgers equation (inverse problem & parameter transfer)

Parameter estimation (×10⁻²):

True = 1.0 PINN = 2.820 PINN‑post = 1.898 Ψ‑NN = 1.465

True = 4.0 PINN = 6.625 PINN‑post = 4.960 Ψ‑NN = 4.084

True = 8.0 PINN = 9.705 PINN‑post = 0.885 Ψ‑NN = 6.673

Ψ‑NN consistently yields the smallest error, and the extracted structure transfers directly to other viscosity settings without modification.

Poisson equation (high‑frequency features & equivariance)

Full‑field L2 error: PINN = 2.633 × 10⁻²; PINN‑post = 2.563 × 10⁻²; Ψ‑NN = 2.464 × 10⁻².

Boundary error is noticeably lower for Ψ‑NN, reflecting the benefit of the discovered permutation‑equivariant structure.

Cylinder flow (cross‑problem structure transfer)

Using the Laplace‑derived symmetric structure for pressure and the Burgers‑derived antisymmetric structure for velocity components, Ψ‑NN attains the lowest L2 errors among all outputs:

Pressure = 7.838 × 10⁻⁴ (≈47 % reduction vs. PINN).

Velocity x = 1.854 × 10⁻⁴.

Velocity y = 1.765 × 10⁻⁵.

Impact and positioning

2019 – PINNs (Raissi): physics via PDE residual loss.

2019 – Hamiltonian NN (Greydanus): manual conservation encoding.

2020 – HFM (Raissi): PDE loss for inverse problems.

2021 – Gradient‑flow analysis (Wang et al.).

2022 – Zhu et al.: hand‑coded symmetry in weight arrangement.

2024 – AsPINN: semi‑automatic symmetry recomposition.

2025 – Ψ‑NN: fully automatic structure discovery and reconstruction.

Ψ‑NN shifts the paradigm from “physics → loss” to “physics → network structure”.

Limitations and future directions

Current theory and experiments are limited to multilayer perceptrons with tanh activations; extending to Transformers or CNNs is non‑trivial.

All PDEs in the study are assumed known; handling unknown or partially known equations will require more sophisticated discovery mechanisms.

The HAC clustering threshold (0.1 in the paper) strongly influences the extracted structure; systematic selection criteria are still open.

Future work may combine Ψ‑NN with Neural Architecture Search, equation‑discovery pipelines, or operator‑learning frameworks such as DeepONet or Fourier Neural Operators.