Harness Evolution vs. Test-Time Scaling: Simple Retries Outperform Complex Self-Improvement

The paper Rethinking the Evaluation of Harness Evolution for Agents (arXiv:2607.12227) from AI2 and the University of Washington investigates whether automated Harness Evolution — where an agent modifies its own prompt, tools, memory, and control logic — truly yields better agent designs or merely benefits from extra inference attempts.

Experimental Setup

Experiments run on Terminal-Bench 2.1 with three models: Claude Opus 4.6, GPT-5.4, and GPT-5.4 mini. All methods start from a minimal harness (only a bash tool, no skills, middleware, or persistent memory). Each task receives a fixed budget of K=5 inference calls. Four methods are compared:

Parallel Sampling — harness unchanged; run the same task independently K times and pick the best result.

Sequential Refinement — harness fixed; each subsequent run sees the previous trajectory and feedback.

Harness Evolution (AHE) — system learns a shared harness from multiple tasks, modifying prompts, tools, and control logic.

Harness Scaling — dynamically adapts the harness for the current task based on its own trajectory.

Results Without Unit Tests (No External Verifier)

Base harness average score: 68.2 .

Parallel Sampling: 72.3 (+4.1)

Sequential Refinement: 69.3 (+1.1)

Harness Scaling: 71.8 (+3.6)

Harness Evolution (AHE): 67.4 (-0.8)

Harness Evolution underperforms even the initial harness. The authors hypothesize that without a reliable verifier, the agent misdiagnoses its own failures, causing harness modifications to optimize based on incorrect signals — making performance worse, not better.

Results With Unit Tests (Reliable Correctness Signal)

All methods improve substantially, but Harness Evolution still does not lead:

Parallel Sampling: average pass@1 = 86.0

Sequential Refinement: average pass@5 = 91.8

Harness Evolution: average pass@1 = 75.8 , pass@5 = 86.2

Harness Scaling: average pass@1 = 82.6 , pass@5 = 89.3

The key metric is pass@1 — if Harness Evolution truly discovered a better general harness, first-try performance should rise markedly. Instead, gains appear only when multiple attempts are allowed, suggesting the improvement comes from test-time compute, not a better harness.

Transfer to Unseen Tasks

To separate task-specific adaptation from general harness improvement, the authors split Terminal-Bench 2.1: 45 tasks for evolution, 10 for validation, 34 held-out test tasks. The evolved harness is evaluated on the unseen 34 tasks. Claude Opus 4.6: 63.3 → 64.5 (+1.2) GPT-5.4: 72.1 → 72.1 (0.0)

Average: 67.7 → 68.3 (+0.6 percentage points)

This tiny transfer gain contrasts sharply with on-training-set improvements, indicating that most of the observed benefit is task-specific adaptation rather than discovery of general agent design principles.

Implications for Agent Evaluation

The paper argues that many modern agent systems (with reflection, memory, planners, multi-agent workflows, self-improvement modules) consume far more inference budget than simple baselines. When a complex system shows a 5-point gain over a minimal agent, a fair comparison must give the minimal agent the same compute budget and feedback. If simple parallel sampling or sequential refinement matches the gain, the complex architecture's algorithmic contribution is questionable.

The authors propose a stricter evaluation standard: compare systems under matched inference budgets and equivalent external feedback . As agents move toward longer trajectories and more complex workflows, this baseline becomes essential.

Resources

Paper: https://arxiv.org/abs/2607.12227 Code: https://github.com/rethinking-harness-evolution Blog:

https://yikee.github.io/harnessevolution/