Measuring and Optimizing Neural Network Simplicity—A Year Before LeCun’s World‑Model Study

The Qianjue Technology and Tsinghua University collaboration proposes a new “complexity ruler” called Effective Degree (ED) that makes the simplicity bias of neural networks both measurable and optimizable. Their work predates a similar world‑model study by Yann LeCun’s group by about one year.

Existing simplicity‑related metrics (parameter norm, loss‑sharpness, Jacobian norm, etc.) often fail to satisfy three desiderata simultaneously: generality across architectures and tasks, quantifiability on real‑scale trained models, and differentiability so the metric can be used during training. To address this, the authors introduce a polynomial‑representation approach that slices a high‑dimensional neural function into many one‑dimensional “function slices” along interpolation paths between data points.

Given two samples x₁ and x₂, they construct the path x(α)=αx₁+(1‑α)x₂, α∈[0,1]. For each path they evaluate the network f and fit the resulting scalar function of α with Chebyshev orthogonal polynomials. The coefficients of the fitted series are weighted by their magnitude to compute an effective degree that reflects how much high‑order content is present. Averaging ED over many data‑driven paths yields a model‑level complexity estimate. The method also incorporates cosine sampling, PCA on high‑dimensional outputs, and data‑distribution‑aware path selection to improve numerical stability.

The authors prove that for multivariate polynomials, random paths drawn from a dense data distribution preserve the algebraic degree ordering, ensuring that the averaging process does not corrupt the complexity ranking.

Empirically, ED predicts generalization gaps far more accurately than traditional proxies. On CIFAR‑10, 27 ResNet‑18 and ViT‑Tiny trainings show a Pearson correlation of 0.99 and an R² of 0.98 between ED and the generalization gap, compared with 0.94/0.88 for the best sharpness metric. In CLIP‑ViT ImageNet fine‑tuning, ED remains positively correlated with the gap while sharpness, adaptive sharpness, and L2 norm exhibit weak or negative correlations.

ED also captures the “grokking” phenomenon: during memorization the metric rises, peaks near the sudden drop in validation loss, and then declines as the model begins to generalize—behaviors that parameter norm and sharpness fail to reveal.

Because polynomial fitting is a least‑squares problem, its coefficients have analytic gradients. The team therefore adds ED as a differentiable regularizer to the training objective, penalizing unnecessary high‑order variation while preserving the primary task loss. Experiments on image classification, text‑image multimodal fine‑tuning, and several reinforcement‑learning environments demonstrate that ED regularization improves in‑distribution accuracy, out‑of‑distribution robustness, and RL performance (see Figure 5).

The paper cautions that “simpler is not always better.” If the simplest features correspond to brittle shortcuts, ED may encourage their use. Moreover, the path‑based polynomial proxy is only an approximation of the full high‑dimensional function and incurs extra computation for sampling interpolation points.

In summary, the Effective Degree provides a practical bridge between theory and practice: it quantifies neural function simplicity in a general, computable, and optimizable way, predicts generalization, tracks training dynamics, and can be directly incorporated into loss functions. Open questions remain about how different architectures and optimizers affect ED dynamics, whether ED can guide checkpoint selection or self‑supervised task design, and whether other function‑space representations can meet the same three desiderata.