From QA to Experiments: How SciAgentGym Puts LLMs into Real Scientific Workflows

DeepMind co‑founder Demis Hassabis views AI as a key driver of scientific discovery, emphasizing its potential to process complex data and uncover hidden patterns.

To move beyond answering questions, scientific agents must handle full research workflows: retrieving databases, invoking specialized software, running computations, analyzing results, and iteratively refining directions based on tool feedback.

Fudan University’s NLP lab therefore released SciAgentGym , a benchmark environment designed for multi‑step scientific tool use. The platform comprises four core components—professional tool libraries, a file system, scientific databases, and a Python interpreter—allowing agents to call tools, execute code, query data, and receive structured feedback, with each task maintaining an isolated execution history.

The design follows three principles:

Type Safety : each tool declares explicit input and output types, enabling the environment to validate calls and ensure compatible tool chaining.

Reproducibility : every tool invocation, intermediate result, and environment feedback is recorded as a structured trace, so evaluation captures the full execution process, not just the final answer.

Extensibility : tools are organized by discipline and standard protocols, making it easy to add new domain‑specific utilities. The authors wrapped mature packages such as RDKit, ASE, SciPy, BioPython, and PyMatGen into categorized tools and screened them with automated unit tests.

Built on this environment, the SciAgentBench benchmark assesses whether current LLM agents can complete long‑range scientific tasks. It contains 259 tasks (1,134 sub‑questions) spanning physics, chemistry, materials science, and life sciences. Tasks are filtered to require genuine tool usage and multi‑modal inputs (≈65% involve molecular structures, spectra, phase diagrams, or experimental images).

Tasks are tiered by difficulty:

L1 : ≤3 steps, testing short tool‑call sequences.

L2 : 4–7 steps, requiring tool composition and intermediate state management.

L3 : ≥8 steps, mirroring real scientific workflows with feedback handling and error correction.

L2 and L3 together account for 79% of the benchmark, emphasizing stability over longer chains. Evaluation metrics include Success Rate (overall task completion) and Success Weighted by Path Length (efficiency, penalizing unnecessary or failed tool calls).

Experimental results show that tool integration raises average success from 23.3% to 28.3%. However, performance degrades on longer tasks: GPT‑5 achieves 41.3% overall, but its L1 success is 58.8% versus 34.6% on L3. Across models, average L1 success is 47.4% while L3 drops to 16.4%, indicating a universal difficulty in maintaining correctness over many steps.

Analysis reveals that frequent tool calls do not guarantee success; many models repeatedly invoke tools without interpreting feedback, leading to repeated errors. Stronger models make fewer calls but leverage intermediate results more effectively, highlighting the importance of understanding environment feedback.

To address the scarcity of high‑quality training data, the authors propose SciForge , which generates executable trajectories by sampling valid tool‑input/output relationships, running them in SciAgentGym, and retaining successful runs (including error‑recovery paths) as training examples.

Training on SciForge data improves agent performance: SciAgent‑8B reaches a 30.1% success rate on SciAgentBench, surpassing the larger Qwen3‑VL‑235B‑Instruct, while SciAgent‑4B climbs to 25.2%.

The authors conclude that, although current LLM agents can use scientific tools, achieving reliable, long‑range workflow execution remains challenging. Future work should focus on teaching agents to interpret feedback, recover from errors, and progressively master open‑ended scientific discovery.