Bengio Team’s ERRLESS Boosts Symbolic Regression with 10× Faster Posterior Sampling

Symbolic regression seeks compact mathematical expressions that explain observed data, but limited and noisy datasets make a single best‑fit formula unreliable because multiple distinct expressions can explain the same observations, and point estimates hide this epistemic uncertainty.

Traditional Bayesian symbolic regression relies on RJMCMC or SMC to sample from the posterior, which requires hand‑crafted proposal distributions and incurs high computational cost. ERRLESS (Bayesian Symbolic Regression with Entropic Reinforcement Learning) reframes the problem as a reinforcement‑learning task, using maximum‑entropy RL to learn a policy that directly generates samples from the joint posterior over expression structure, constants, and observation noise.

The core posterior modeled by ERRLESS treats the expression tree, its numeric constants, and the noise variance as a single stochastic object. A bottom‑up (post‑order) generation process is adopted: at each step the operator, its operands, and dimensional constraints are already known, allowing the model to enforce arity, dimensionality, and structural constraints during generation. The policy network follows a Transformer‑style architecture with three heads that output (i) logits for the next action, (ii) parameters of a Gaussian‑mixture distribution for constants, and (iii) parameters of a log‑normal‑mixture distribution for noise, guaranteeing a positive noise standard deviation.

Training uses the unnormalized log‑density of the posterior as the reward and adds an entropy term so that the optimal policy minimizes the KL divergence between its distribution and the target posterior. Optimization is performed with the GFlowNet trajectory‑balance objective, enabling off‑policy learning that approximates the desired posterior distribution.

Empirical evaluation on seven small synthetic tasks shows that ERRLESS produces stable posterior predictions, whereas the SMC‑based baseline PySIPS, while yielding stronger individual expressions, fails to provide reliable posterior estimates and often diverges outside the training interval. On the Feynman symbolic‑regression benchmark (116 physical formulas), ERRLESS attains an AUC of 0.924 , surpassing DSR (0.873), PhySO (0.893) and BSR (0.702), and its performance remains virtually unchanged when target noise varies from 0.001 to 0.1.

On a more challenging Blackbox benchmark, ERRLESS achieves an AUC of 0.350 , outperforming BSR (0.19) and AIFeynman (0.004) but trailing XGBoost (0.625). In terms of efficiency, ERRLESS runs roughly an order of magnitude faster than most learned symbolic‑regression methods, because it integrates constant optimization into the sampling process and avoids a separate post‑hoc fitting step.

The method incorporates an explicit unigram prior derived from the Encyclopaedia Inflationaris, which biases the search toward shorter, physically plausible expressions when data are insufficient. As more data become available, the likelihood term increasingly dominates the posterior, reducing the prior’s influence.

Limitations include a performance drop on very complex target expressions and the need to retrain the sampler for each new dataset. Future work aims to endow the sampler with cross‑dataset generalization, eliminate per‑dataset retraining, and extend the approach to operator‑library learning and the discovery of ODE/PDE symbolic forms.