Can AI Run an Entire Research Project End‑to‑End? Inside the AiScientist Breakthrough

1. The Core Question of the Paper

The authors shift focus from "Can AI do a single research step?" to "Can AI sustain an entire research engineering pipeline, continuously advancing from paper understanding to implementation, experimentation, debugging, and iterative improvement?" This long‑horizon challenge involves tightly coupled stages that must remain coherent over many cycles.

2. Why Machine‑Learning Research Engineering Is Hard

Research engineering is not a single task but a chain of stages: understanding the paper, configuring environments, handling dependencies, preparing data, implementing models, running experiments, reading logs, diagnosing failures, and iterating. Each stage is underspecified, burdensome to set up, provides delayed/confounded feedback, and suffers from state continuity problems.

Long‑horizon research engineering is a systems problem, not merely a collection of local issues.

3. How AiScientist Tackles the Problem

AiScientist adopts a "thin control, thick state" philosophy. The top‑level orchestrator maintains only high‑level phase control—knowing which stage comes next and which component should handle it—while the detailed project state is stored in a shared workspace that all agents can read and write.

Two key mechanisms enable this:

Hierarchical orchestration : A top‑level orchestrator plans and advances phases, delegating paper analysis, task planning, code generation, experiment execution, and debugging to specialized agents.

File‑as‑Bus : Critical artifacts (paper analysis, plans, code, logs, diagnostics) are continuously written to a shared file system that serves as the system of record, allowing subsequent agents to pick up the exact state left by previous steps.

This design ensures that each iteration builds on concrete, durable evidence rather than vague conversational summaries.

4. Empirical Results

On the MLE‑Bench Lite "Detecting Insults" task, AiScientist completed 74 experiment cycles in 23 hours, raising validation AUC from 0.903 to 0.982 and achieving 81.82% Any‑Medal on the benchmark. On PaperBench, it improved the best‑matching baseline by an average of 10.54 points.

These results show that AiScientist does not merely produce a one‑off demo; it continuously pushes the research pipeline forward, maintaining a stable chain of improvements.

5. Three Takeaways

Takeaway 1: Sustained multi‑stage capability matters more than isolated strength.

AiScientist’s advantage on both PaperBench (end‑to‑end system building) and MLE‑Bench Lite (iterative solution refinement) demonstrates that the system can orchestrate several hard stages into a cohesive, advancing process.

Takeaway 2: More interaction alone is insufficient; each round must reliably inherit prior evidence.

Experiments comparing AiScientist with IterativeAgent show that simply adding more interaction does not yield better performance unless the system can preserve and exploit the state generated in earlier rounds.

Takeaway 3: The File‑as‑Bus abstraction raises the ceiling for later‑stage refinement.

Ablation studies reveal that removing File‑as‑Bus drops performance by 6.41 points on PaperBench and 31.82 percentage points on Any‑Medal, especially hurting higher‑tier metrics (Silver, Gold, etc.). This indicates that durable state storage improves not only executability but also result fidelity.

6. Final Thoughts

AiScientist illustrates that truly long‑horizon AI research agents must combine reasoning, code generation, tool use, and robust state management. While many agents can write code or draft papers, only systems that reliably persist evolving project state can consistently win strict benchmarks.

Toward Autonomous Long-Horizon Engineering for ML Research
https://arxiv.org/pdf/2604.13018
https://github.com/AweAI-Team/AiScientist