How Autoregressive Boltzmann Generators Are Redefining Molecular Sampling

Background

For a molecular system with potential energy \(E(x)\), the equilibrium distribution is the Boltzmann distribution \(p(x) \propto \exp(-E(x)/kT)\). Direct sampling from \(p(x)\) is expensive because molecular dynamics (MD) requires many energy evaluations and produces highly correlated samples, especially for systems with multiple metastable basins or high energy barriers.

Boltzmann Generators aim to learn a generative model \(q_{\theta}(x)\) that can produce near‑equilibrium conformations. When the model provides an exact likelihood, importance‑sampling weights \(w(x)=p(x)/q_{\theta}(x)\) can correct any residual bias, combining fast generation with physical accuracy.

Core Idea: Autoregressive Decomposition

The paper replaces the traditional normalizing‑flow backbone with an autoregressive model. The joint distribution of molecular coordinates \(x\) is factorised as q_θ(x)=\prod_j q_θ(x_j\mid x_{<j}), where each coordinate (or its discretised version) is conditioned on all previously generated coordinates. This removes the need for a globally invertible mapping and avoids topological constraints of flows.

Flexibility : No diffeomorphism or topology preservation limits, allowing probability mass to be allocated across many conformational basins.

Direct likelihood : The log‑likelihood is obtained by a single forward pass (or sequential conditional predictions), simplifying importance‑sampling weight computation.

Scalability : The formulation can leverage large‑scale Transformer architectures and token‑level likelihood techniques developed for language models.

Method: Generation and Correction Pipeline

Represent a conformation as an ordered set of variables. Continuous coordinates are discretised into a fixed number of bins; the model predicts the bin for each dimension and adds intra‑bin noise to reconstruct a continuous value.

Train a Transformer‑style autoregressive network by maximising the likelihood of training conformations, i.e. predicting each coordinate conditioned on its prefix.

During generation, sample coordinates sequentially to obtain a proposal distribution \(q_{\theta}(x)\). The raw proposal already yields near‑equilibrium samples.

Compute importance‑sampling weights \(w(x)=p(x)/q_{\theta}(x)\) using the physical energy function and the model likelihood, then apply self‑normalized importance sampling (SNIS) or sequential Monte Carlo (SMC) for resampling.

Inference‑time Interventions

Because generation proceeds step‑by‑step, the temperature of the sampling distribution can be tuned. A temperature that is too low concentrates probability in high‑likelihood, low‑energy regions and reduces diversity; a temperature that is too high spreads probability into irrelevant regions and degrades sample quality. Experiments on the AL3 peptide find an optimal temperature of approximately \(T=1.02\).

SNIS reweights samples with the exact proposal likelihood. For larger or more complex systems the paper explores Twisted SMC variants that gradually introduce the target distribution to mitigate weight degeneration.

Experiment 1: Single‑Peptide Benchmarks

ArBG is evaluated on four peptides: AL3, AL4, AL6 and the ten‑residue mini‑protein Chignolin. It is compared against flow‑based Boltzmann Generators, GMM‑PixelCNN++, BG Prose and GIVT. Three Wasserstein‑2 metrics are reported:

E‑W2 : distance between generated and reference energy distributions.

T‑W2 : distance between generated and reference torsion‑angle distributions.

TICA‑W2 : distance in the slow‑mode (time‑structure based independent component analysis) space.

Under a budget of \(2\times10^{5}\) energy evaluations, ArBG achieves on the challenging Chignolin system: E‑W2 = 1.723, T‑W2 = 2.632. By contrast, the flow‑based SBG reports \(E‑W2 = 10.819\) and GIVT reports \(E‑W2 = 45.646\). Energy histograms and Ramachandran plots show that the ArBG proposal already covers the main energy basins and that SNIS resampling brings the distribution close to the MD reference.

Experiment 2: Transferable Model Robin

A 132‑million‑parameter model named Robin extends the ArBG framework with sequence‑conditional conditioning, enabling zero‑shot generation for unseen peptide sequences. On the ManyPeptidesMD dataset, Robin attains \(E‑W2 = 3.615\) for eight‑residue peptides with the same \(2\times10^{5}\) evaluation budget, a 61 % reduction compared with BG Prose (\(E‑W2 = 9.360\)). T‑W2 and TICA‑W2 remain competitive, indicating accurate coverage of both energy and slow‑mode statistics. Visualization of the octapeptide CGSWHKQR shows strong overlap between Robin’s TICA distribution and the MD reference. Ablation Study: Discretisation Resolution and Sampling Temperature Increasing the number of discretisation bins reduces quantisation error and improves post‑resampling \(E‑W2\), at the cost of higher modelling complexity. Temperature exhibits a U‑shaped curve: too low compresses diversity, too high harms quality. The optimal temperature (~1.02) demonstrates that the best proposal for importance sampling is not necessarily the raw model distribution. Relation to Prior Work Boltzmann Generators: earlier work relied on normalizing flows; ArBG replaces the flow backbone with an autoregressive model. Molecular generative models: many Transformer or diffusion models generate valid molecules but lack exact likelihoods for equilibrium sampling; ArBG provides both generation and likelihood. Autoregressive continuous‑space modelling: techniques such as PixelCNN and mixture‑density networks inspire the conditional decomposition used in ArBG. Conclusion Autoregressive Boltzmann Generators show that self‑attention‑based autoregressive models can serve as a viable backbone for equilibrium molecular sampling, offering greater flexibility, simpler likelihood computation, and natural inference‑time controls. The Robin model demonstrates that a single large‑scale autoregressive model can transfer across peptide sequences, moving Boltzmann Generators toward a more generalizable sampling framework.