How Autoresearch Achieves 5‑Minute Self‑Evolving AI Research Loops

Autoresearch is an open‑source framework released by AI pioneer Andrej Karpathy that enables a fully autonomous AI research cycle. The entire system consists of only 630 lines of code, runs on a single GPU, and in internal tests a 0.8 B model gained a 19 % performance boost within eight hours, prompting even the Shopify CEO to praise it.

The core concept is to automate the iterative trial‑and‑error loop of model training. Each experiment is constrained to a fixed five‑minute training time to remove duration variance, and the only evaluation metric is val_bpb, which correlates lower values with better model quality and is independent of model size. This design compresses the training logic into a lightweight, fast‑running process.

The codebase is organized into three clear files. prepare.py is read‑only and defines constant training parameters, downloads the dataset, adapts the tokenizer, and supplies experimental utilities. train.py is the sole file the AI may edit; it contains the GPT architecture, optimizer, and training‑loop logic, allowing the AI to adjust layer count, learning rate, and other hyper‑parameters while keeping changes traceable. program.md is authored by a human and records research directions, experiment rules, and baseline commands; changing research focus only requires editing this markdown file.

During execution, the AI follows the instructions in program.md, makes one to two targeted edits in train.py, and launches a five‑minute training run. After training, the framework automatically scores the result with val_bpb. If the score improves (i.e., decreases), the modification is retained as the new baseline; otherwise the change is discarded and the previous best version is restored. The next experiment then begins immediately. At five minutes per round, the system can complete more than ten experiments per hour; in a test of roughly 250 rounds the AI identified 29 genuinely beneficial optimizations while also collecting failed attempts as negative knowledge.

Karpathy outlines a future roadmap that moves beyond simulating a single PhD student. He proposes leveraging the “large‑scale, distributed, asynchronous” model of projects like SETI@home to build an AI research ecosystem that mirrors an entire community of doctoral researchers. He also notes that the current linear master‑branch workflow and Git/GitHub version‑control model impose structural limits on fully automated research.

GitHub: https://github.com/karpathy/autoresearch