ChemGraph: 13 Benchmarks Reveal LLM Agent’s Capabilities in Computational Chemistry

Recent advances in large language models (LLMs) have enabled a new generation of scientific agents. Prior work such as ChemCrow, CACTUS, and MDCrow demonstrates LLM‑driven assistance for synthesis planning, property prediction, and molecular dynamics. However, computational chemistry lacks a unified benchmark for assessing such agents.

To fill this gap, researchers at Argonne National Laboratory propose ChemGraph, an LLM‑driven agent framework built on LangGraph and following the ReAct paradigm. ChemGraph orchestrates tool calls through three core components:

State : a shared data structure representing the current execution context.

Nodes : Python functions (including LLM calls) that receive the state and return an updated state.

Edges : functions that direct the message flow, either as fixed directed edges or conditional edges that choose the next node based on runtime conditions.

Using these components, ChemGraph can translate SMILES strings and molecular structures into geometry optimizations, vibrational analyses, and thermochemical calculations, supporting semi‑empirical methods, machine‑learning potentials, and density‑functional theory (DFT). The system runs inside Docker containers for safety.

The authors designed 13 benchmark experiments—360 individual evaluations—covering three input categories (molecule name, SMILES string, chemical reaction). The first 11 tasks require up to four tool calls each, while the final two (react2enthalpy and react2gibbs) demand 9–12 calls to compute reaction thermochemistry.

Single‑agent evaluation used GPT‑4o‑mini, Claude‑3.5‑haiku, Qwen‑2.5‑14B, and (for the two thermochemistry tasks) GPT‑4o. Results show that on the name‑to‑SMILES task Claude‑3.5‑haiku sometimes bypasses the molecule_name_to_smiles tool, leading to the lowest average tool‑call count, while GPT‑4o‑mini consistently produces correct SMILES. For name‑to‑XYZ, name‑to‑opt, and name‑to‑vib tasks, both GPT‑4o‑mini and Claude achieve >83 % accuracy. In the smi2xyz‑to‑smi2file chain, Claude and GPT‑4o‑mini exceed 83 % accuracy, with Qwen’s minimum at 77 %.

For the two reaction‑thermochemistry tasks, Qwen‑2.5‑14B performs poorly (<20 % accuracy). GPT‑4o‑mini attains 40 % (enthalpy) and 49 % (Gibbs), Claude‑3.5‑haiku reaches 67 % and 69 %, and GPT‑4o exceeds 83 % on both.

Multi‑agent evaluation adds additional LLMs to the same tasks. The multi‑agent configuration substantially raises accuracy: GPT‑4o‑mini improves from 40 % to 87 % (enthalpy) and from 49 % to 87 % (Gibbs); Claude‑3.5‑haiku rises from 67 % to 87 % and from 69 % to 93 %; GPT‑4o achieves perfect 100 % success on both tasks. Qwen‑2.5‑14B shows only limited gains due to frequent tool‑call errors.

The study, titled “ChemGraph as an agentic framework for computational chemistry workflows” and published in Nature Communications Chemistry , concludes that ChemGraph demonstrates the promise of AI‑driven automation for molecular simulation workflows. The authors plan to integrate more tools, refine the agent architecture, and enhance high‑performance‑computing support, while emphasizing that ChemGraph complements rather than replaces traditional computational chemistry software.