Beyond AutoResearch: Co‑Evolving Strategies and Harnesses with EvoTrainer for Autonomous Training Loops

EvoTrainer is a self‑evolving training method that jointly evolves large‑language‑model (LLM) policies and the Training Harness, turning the training decision system itself into an optimizable component.

Why static harnesses fail in Agentic RL

Traditional autonomous training treats the Harness as a fixed infrastructure—reward definition, sample filtering, back‑testing logic, and diagnostics are set once and never change. As the policy grows stronger, this assumption breaks down: the original reward loses discrimination, existing evaluation cannot spot speculative behavior, coarse diagnostics miss new failure modes, and static sample‑filtering no longer supplies high‑information signals.

Co‑evolutionary loop

The authors propose a closed‑loop where each policy improvement exposes new bottlenecks; the Harness perceives these bottlenecks, upgrades its diagnostics and back‑testing capabilities, and then provides more reliable optimization signals for the next policy update. EvoTrainer systematizes this loop at training time rather than only at inference.

Related work

AutoResearch demonstrates that agents can automate the experiment cycle (code modification → training → metric verification → keep/rollback), but it still relies on a static Harness and a “fast‑trial‑and‑error” mindset based on a single validation score. Meta‑Harness and AHE show that Harnesses can be optimized at inference time, yet their evolution is confined to Inference‑Time and does not address the training‑time signal gap.

Core contributions of EvoTrainer

Unifies three strands—AutoResearch‑style autonomous training, Meta‑Harness dynamic harness evolution, and AHE’s process‑evidence‑driven improvement—into a single training‑time co‑evolution framework.

Introduces Memory & Skill : each training round is recorded, failure diagnostics are stored, and reusable analyzer skills , repair strategies , and procedure templates are built.

Implements a version chain: every policy update creates a new version; the Harness adjusts its diagnostics based on the newly observed bottlenecks. When evidence is insufficient, the Harness upgrades itself instead of blindly launching the next experiment.

Upgrades include new behavior analyzers, targeted back‑testing cases, retrieval of similar historical diagnostics, and dynamic sample‑filter adjustments.

Reward redesign for multi‑step tasks

Relying solely on the final test‑case score (a static reward) fails for long‑trajectory tasks such as Software‑Engineering (SWE) RL. EvoTrainer adds behavior rewards that capture “edit‑verify” loops and adjusts their relative weights (e.g., giving the Edit‑Verify Token (ETT) a slightly higher coefficient than the pre‑edit behavior (SBE)) to increase ranking power. It also filters rollout groups that lack discriminative signal, ensuring GRPO receives stable relative‑learning cues.

Experimental validation

The system is evaluated on three task families covering single‑step inference, execution feedback, and long‑term tool use:

Math : benefits mainly from increased computational capacity.

Coding : gains come from reward shaping and sample filtering.

SWE : requires behavior‑sensitive training paths and the new process signals.

Across all three families EvoTrainer consistently improves performance, demonstrating that the gains stem from task‑specific harness upgrades rather than a one‑size‑fits‑all configuration.

Limitations and future work

High inference, retrieval, and back‑testing computational cost.

Current experiments contain relatively short evolution trajectories.

Performance still depends on a strong base LLM.

Future efforts will focus on reducing overhead, extending the length of evolution chains, and making the training decision system cheaper, more stable, and more portable.