HarnessPro: Pareto-Optimized Recursive Self-Improvement Cuts Agent Costs While Boosting Performance

Recursive self-improvement (RSI) methods like RHO can significantly boost agent pass rates on coding benchmarks, but they often increase per-task execution costs — RHO raised cost by 42.6% (from $1.88 to $2.68) while lifting pass rate from 59.0% to 80.0% on SWE-Bench Pro with GPT-5.5 + Codex. HarnessPro addresses this performance–cost dilemma by treating pass rate and execution cost as dual optimization objectives and using Pareto optimization to retain multiple candidate harnesses per evolution round.

Four-Agent Collaborative Evolution

HarnessPro employs four specialized agents: a Trajectory Analyzer that summarizes failure causes and redundant overhead from execution traces; an Evolution Planner that proposes three improvement directions (cost-first, balanced, performance-first) based on diagnostics and cross-candidate code diffs; a Harness Editor that implements changes in isolated workspaces; and a Test Generation Agent that writes 10 executable tests per training/validation task from the task specification and initial trajectory. Tests are frozen after generation so all candidates are evaluated on the same suite.

Evolution Loop and Pareto Selection

Each benchmark first selects 10 diverse training tasks via Determinantal Point Process (DPP). In every evolution round, each retained harness runs 3 times per training task. The Trajectory Analyzer aggregates results, the Planner proposes three variants per harness, and the Editor produces new candidates. Candidates are filtered on a validation set: any candidate with pass rate below the initial harness is discarded; then Pareto dominance removes any candidate that is no better in both pass rate and cost. Remaining candidates are scored by normalized weighted squared distance to the ideal (max pass rate, min cost) with weights (1,2) for Lite, (1,1) for Medium, (2,1) for Heavy. Up to three harnesses (one per tier) advance to the next round.

Test Generation vs. LLM-as-Judge

RHO relies on an LLM to judge trajectory quality, which the authors found overly optimistic — detecting only 5 of 12 real failures (41.7% recall). HarnessPro's generated tests caught 10 of 12 (83.3% recall). Ablation removing the test generator dropped pass rates by 3.4–10.0 percentage points across tiers and benchmarks. Final reported numbers always come from the held-out test set with official evaluation.

Experimental Results

Experiments ran on Terminal-Bench 2, SWE-Bench Pro, and ProgramBench using GPT-5.5 + Codex and GLM-5.2 + Claude Code. Baselines included RHO, Dynamic Cheatsheet, and RHO+Iteration (same evolution rounds with cost-aware prompts).

Heavy : highest pass rate on all three benchmarks with GPT-5.5, and lower cost than RHO. On Terminal-Bench 2: 80.8% pass rate (+4.5 pp) at $0.72 (−33.9%); SWE-Bench Pro: 80.0% (+0 pp) at $1.98 (−26.1%); ProgramBench: 69.6% (+0.8 pp) at $2.95 (−17.8%). Execution time on SWE-Bench Pro fell from 520 s to 412 s.

Medium : balances both metrics. SWE-Bench Pro: 79.0% (−1 pp vs RHO) at $1.26 (−53.0%). On Terminal-Bench 2, Medium matched RHO+Iteration pass rate at 31.7% lower cost.

Lite : minimizes cost while exceeding original Harness pass rate. Cost reductions of 17.7%–57.4% across benchmarks.

Results with GLM-5.2 + Claude Code showed the same pattern: Heavy beat RHO on pass rate and cost across all benchmarks.

Evolved Cost-Saving Rules

The evolved harnesses learn dynamic, task-adaptive rules rather than static truncation. On Terminal-Bench 2 (Medium), the harness selects a standard execution config and inserts runtime hooks: at 14 tool calls it reminds the agent to batch calls and avoid duplicate checks; at 26 calls it urges transition to final verification; at 44 calls it blocks further tool use. Hooks are appended to tool results, preserving the prompt prefix to exploit cached-input pricing (cached input priced at 1/10 of uncached). The harness also applies RTK compression to deduplicate logs and trim long outputs. ProgramBench Medium uses fixed high reasoning effort but allocates budget to reference-program comparison and offline reconstruction. Ablations showed that simply adding dynamic routing or budget hooks to RHO reduced cost but dropped SWE-Bench Pro pass rate to 76.0% and 75.0% respectively, while full HarnessPro Medium kept 79.0%.

Cost per Successful Task (CoP)

When failed-task overhead is amortized over successes, Lite achieves $1.11 CoP on SWE-Bench Pro versus RHO's $3.35 — a 3× improvement in cost efficiency.

Conclusion

HarnessPro demonstrates that recursive self-improvement can be made cost-aware by combining unlabeled feedback, whole-harness recursive editing, Pareto multi-objective optimization, and executable test-based validation. The evolved harnesses internalize a pragmatic principle: maximize tasks completed per dollar spent. Code and artifacts are open-sourced at https://github.com/pzs19/HarnessPro and https://pzs19.github.io/HarnessPro/.