MolRSI: Self-Improving AI Agent Doubles Wet-Lab Hit Rates in Molecular Discovery

Dual-Loop Architecture for Recursive Self-Improvement

MolRSI (Molecular Recursive Self-Improving agent) frames the design of molecular discovery workflows as a search problem. It employs two nested loops:

Inner loop : Optimizes the workflow for the current task using experience-guided Monte Carlo tree search (MCTS). Each candidate workflow is a node; modifications create new branches. Evaluation of the current task, historical performance on related tasks, and exploration of new proposals jointly guide the next step.

Outer loop : Accumulates and reuses experience across tasks. Three memory stores — scheme memory (workflow performance), improvement memory (modification effects), and execution memory (runtime experience) — are carried forward to new tasks, providing better starting points and search policies.

The system converts methods from papers and code into verified executable skills, then combines, tests, and modifies workflows. After each attempt it records not only a score but also the conditions under which the workflow succeeded, which modifications helped, and how errors were fixed. This creates a recursive cycle: search produces experience, experience changes the search policy, and new search yields new experience.

Cross-Task Experience Transfer Boosts Early Performance

To quantify the value of transferred experience, the team first let MolRSI accumulate experience on 25 molecular property prediction and cytotoxicity tasks, then evaluated it on 6 held-out antimicrobial prediction tasks. Both groups shared the same initial skill library, base models, and search budget, and both could update memory within the current task; only one group inherited cross-task experience.

With cross-task experience, the average test AUPRC (area under the precision-recall curve) improved across all 6 antimicrobial tasks. AUPRC measures the ability to rank active molecules early in the candidate list.

On E. coli and N. gonorrhoeae tasks, MolRSI's very first evaluated workflow (zero-search) already outperformed the final workflows selected by Biomni, STELLA, Codex, Claude Code, and Claude Science after 20 search rounds.

Retrospective analysis expanding the source task pool from 2 to 25 tasks showed that the average test AUPRC of transferable workflows rose from ~0.32 to ~0.37, surpassing the 0.361 achieved by a system with no prior experience after 20 search rounds.

This demonstrates that more diverse experience populates the method library with better workflows, giving new tasks a stronger starting point.

Antimicrobial Screening: 15.9% Wet-Lab Hit Rate vs. 8.7% for Human Expert Workflow

MolRSI was tasked with screening ~12.1 million compounds for activity against methicillin-sensitive Staphylococcus aureus (MSSA) while balancing cytotoxicity and structural novelty. The optimized workflow:

Filters for structural novelty, removing common antibiotic scaffolds and compounds too similar to known antibiotics.

Predicts toxicity against three human cell lines.

Ranks activity using an ensemble of two MiniMol variants (55% and 30% weight) and LANTERN–RADR (15% weight).

Clusters structurally similar molecules and picks candidates from different clusters to preserve chemical diversity.

From the library, 107 compounds were selected for experimental testing, yielding 17 hits (MSSA MIC ≤ 32 μg/mL), a hit rate of 15.9% . The previous human-expert-designed non-agentic workflow tested 241 compounds and obtained 21 hits (8.7% hit rate) using the same training data, library, and hit definition. This represents a relative improvement of ~82% .

On an independent test set of 271 compounds, MolRSI's activity predictor also outperformed the expert workflow's predictor (AUROC and AUPRC 0.300 vs. 0.090). Two representative hits (C3, C32) showed MICs of 1 and 8 μg/mL against MRSA, with cytotoxicity concentrations higher than their antibacterial concentrations, confirming a favorable therapeutic window.

De Novo pMHC Binder Design: 8 Specific Binders vs. 0 for Claude Science

A second, independent challenge required designing from scratch proteins that bind specifically to peptide–MHC (pMHC) complexes. The difficulty: target and off-target pMHCs share the same MHC framework, differing only in a few exposed peptide residues. The workflow must both assess binding confidence and discriminate between highly similar targets.

MolRSI first accumulated experience on 14 small-protein binder ranking tasks, then developed a pMHC ranking pipeline using two gp100 datasets (MART-1 held out for testing). With prior experience, enrichment factors at top 1% and 2% reached 3.65 and 5.55; without it, both were 0.

The discovered ranking pipeline uses ESMFold2-Fast structure prediction followed by eight equally weighted scores covering interface contacts, complex/interface confidence, monomer folding confidence, and helical features. Scores are normalized to "larger is better" and summed. Specificity is enforced by paired evaluation: each candidate binder is scored against both target and off-target pMHC (same MHC, swapped peptide). Candidates must have target score > off-target score and pass interface confidence and monomer quality thresholds, then are ranked by target score.

With the ranking pipeline fixed, MolRSI searched over generation methods (backbone generation, peptide hotspot filtering, SolubleMPNN sequence design) for 30 rounds, selecting the top three workflows, scaling generation, merging, deduplicating, and filtering. In prospective experimental validation via two rounds of phage display:

MART-1 target: 5 specific binders out of 251 evaluable designs.

SARS-CoV-1 target: 3 specific binders out of 266 evaluable designs.

Claude Science (same targets, constraints, compute budget): 0/288 and 0/267.

Combined: 8 specific binders vs. 0 .

Experimental criterion: target enrichment log₂FC > 4 and higher than the maximum enrichment among all measured off-targets for that design. An ablation removing MolRSI's ranking step yielded zero specific hits from 479 evaluable designs, confirming that candidate ranking is critical for experimental success. Predicted structures of representative hits show the binder positioned over the exposed peptide, contacting multiple variant residues, consistent with the design goal.

Key Technical Details

Search algorithm : Experience-guided MCTS with scheme, improvement, and execution memories.

Antimicrobial workflow components : Novelty filter, three-cell-line toxicity prediction, MiniMol (×2) + LANTERN–RADR ensemble (55/30/15), diversity-preserving clustering.

pMHC workflow components : ESMFold2-Fast, eight scoring metrics (interface contacts, complex/interface pLDDT, monomer pLDDT, helical content, etc.), paired target/off-target evaluation, SolubleMPNN for sequence design.

Compute budget parity : All comparisons (human expert, Claude Science, other baselines) used identical training data, compound libraries, hit definitions, design constraints, and computational budgets.

Limitations and Future Work

The current recursive improvement operates primarily in computational search; future work aims to incorporate wet-lab feedback directly into the outer loop. Quantitative affinity measurements and broader cross-reactivity profiling for the pMHC binders remain to be done. The approach demonstrates that letting one research project leave behind reusable workflows and search experience for the next project can materially improve real-world experimental outcomes.

References

Paper: MolRSI: A Recursive Self-Improving AI agent for Molecular Discovery (arXiv:2610.39714)

Project website: https://yikunpku.github.io/MolRSI/