How Autoresearch Turns Your Team into a Self‑Improving Research Engine

Introduction

Many teams have clever people and ideas, but they often lack a sustainable experiment mechanism. The author argues that the scarcity is an executable autonomous loop that can keep research running continuously.

Core Concept

The key idea is succinctly captured as "people write rules, agents run experiments" and memorized with the mantra “one changeable point, fixed budget, unchanged evaluation, roll‑backable result.”

Autonomous Experiment Loop (AEL) Elements

Set Boundaries : Only the optimization target may be changed, producing an allow/deny list.

Fixed Budget : Each round has a fixed cost—e.g., 5 minutes, N runs, token limit.

Fixed Evaluation : The evaluator is read‑only, outputting a score/compliance metric.

Record & Compare : Each round logs results (tables, logs).

Selective Rollback : Keep the best outcome; otherwise revert.

What Makes Autoresearch Special?

1. Single Mutable File

Only train.py may be edited by the agent; all other files, especially the evaluator, act as a “constitution,” ensuring controllability, auditability, and safe rollback.

2. Fixed 5‑Minute Budget

The budget is not about saving time but about preventing unfair advantage . By fixing the cost per round, larger models or longer batches cannot dominate automatically; effectiveness is measured fairly.

3. Immutable Evaluator

The evaluator resides in prepare.py and is explicitly prohibited from modification, establishing a hard rule that you can only improve the subject, not the judge.

4. Logging & Rollback

Each experiment writes a result row (commit, metric, notes). Worse results are rolled back, turning trial‑and‑error into reusable data rather than waste.

Four Levers to Adopt Anywhere

Lever A – Single Changeable Surface : Limit modifications to one file (e.g., train.py) or one content area (title/structure/body) while keeping the scorer immutable.

Lever B – Fixed Budget : Enforce a constant time or token budget per round (e.g., 5 minutes, 3–6 rounds, 2000 tokens).

Lever C – Fixed Evaluator : Keep the scoring/compliance script read‑only (e.g., evaluate_bpb).

Lever D – Record + Rollback : Store each round’s score.json, compliance.json, and change summary; retain the best version.

Mapping the Design to Other Teams

The table in the original article is rendered as bullet points:

Single Changeable Surface : Edit only train.py; analogous to editing only the body of a customer‑service script.

Evaluator Lock : prepare.py is read‑only; similar to a fixed quality‑check checklist.

Fixed Budget : 5‑minute training per round; comparable to a fixed number of runs or token limit.

Single Primary Metric : val_bpb (lower is better); can be replaced by conversion rate, pass rate, etc.

Record + Rollback : Log version, score, reason; keep the best.

Practical Examples

Three concrete scenarios illustrate how to apply the loop:

Content Creation Revision : Changeable parts are title/structure/body; evaluator checks clarity, depth, CTA; budget of 3–6 rounds; output includes per‑round scores and the final best version.

Customer‑Service Script Optimization : Editable text; evaluator checks compliance and key information; fixed sample size per round; output shows pass‑rate curve and best script.

Code/Workflow Regression Optimization : Editable module or config (e.g., cache policy, concurrency, prompt template); evaluator runs a fixed regression suite; budget limits time or case count; output records metric changes and best version.

Learning Path

Step 1 – Read the README : Identify the three minimal components – prepare.py (fixed evaluator), train.py (only mutable file), and program.md (research policy).

Step 2 – Deep Dive into program.md : Learn to write a formalized process covering setup, loop execution, and crash handling.

Step 3 – Lock the Evaluator : Ensure the evaluator cannot be altered; otherwise, any optimization is meaningless.

Implementation Checklist (Content‑Version Example)

Write a brief: target audience, pain point, three takeaways.

Define gate‑keeping rules: structure (6 sub‑headings), mandatory tables/lists, paragraph length ≤ 3 lines, bold key points.

Include a CTA at the end.

Enforce compliance: forbid prohibited expressions.

Common Pitfalls

Chasing scores without delivering value – ensure the scorer covers density, executability, credibility.

Allowing changes to the scorer – leads to cheating and drift.

Missing termination conditions – the system may run forever and exhaust the budget.

No human audit – always keep a safety valve.

One‑Page Summary for Stakeholders

The value of autoresearch lies not in model training but in codifying research organization as a set of enforceable rules . The four levers—single mutable surface, fixed budget, immutable evaluator, and systematic logging/rollback—enable any team to turn their work into an autonomous experiment loop.