How Can Agents Learn to Train Models? From Score‑Chasing to Verifiable Self‑Evolution

Problem

Automatic post‑training pipelines often modify data, hyper‑parameters, inference settings and evaluation pipelines in a single step. When a score improves it is unclear which change caused the improvement and the process cannot be reproduced. Recursive self‑improvement (RSI) therefore requires a closed research loop: read failure evidence, hypothesize missing capabilities, design data experiments, train and select checkpoints, interpret results and decide the next action.

RSIBench‑Data Benchmark

RSIBench‑Data turns the above issue into a controlled scientific experiment. It fixes the model, training backend, checkpoint serving, evaluation sandbox, task subset, budget and scoring protocol, leaving the agent to decide only the data‑research actions. The benchmark records four state‑of‑the‑art researcher systems, six task categories and 24 full research trajectories.

Research Questions Enabled

Can the agent detect genuine capability gaps from failure samples?

Can it design experiments that provide real information gain rather than merely generating more data?

Can it decide when to keep or discard checkpoints, when to stop an experiment, roll back, or change research direction?

Can the system trace which research decision caused a performance change?

Key Findings

All four evaluated agents are able to take part in research, but “proposing experiments” does not equal “completing research.” Stable research ability depends on a complete feedback loop—especially failure diagnosis, checkpoint management, stop‑criteria and rollback—rather than on producing additional training data.

An exploratory RSI run driven by the Kimi K2.6 researcher harness performed seven LoRA experiments. The candidate‑checkpoint pass rate rose from 8 % to 22 %, demonstrating that RSIBench‑Data can surface measurable gains.

Conclusion

RSIBench‑Data provides a reproducible research problem instead of a single leaderboard score. It makes the research process observable, the sources of improvement explainable, and failures usable as scientific evidence, thereby advancing verifiable self‑evolving AI agents.