HarnessPro: Self-Improving Agents Cut Execution Costs 26% While Raising Pass Rates

HarnessPro: Recursive Self-Improvement for LLM Agents

On October 19, researchers from Tsinghua University, Zhipu AI, Shanghai Jiao Tong University, University of Warwick, and Kuaishou published HarnessPro , a system that lets an agent review its own execution records, detect failures and waste, and automatically modify its reusable operating framework. The process iterates: execute, analyze, plan, edit, evaluate, and repeat.

Core Loop: Analyze, Plan, Edit, Evaluate

Trajectory Analysis Agent examines tool calls, environment feedback, and token consumption from multiple runs of the same task to identify failure causes, redundant searches, and unnecessary checks.

Planning Agent produces three modification proposals biased toward cost saving, balanced trade-off, or performance priority.

Framework Editing Agent implements each proposal in an isolated workspace, with changes tracked via Git for traceability and rollback.

The framework modifications target reusable work patterns — e.g., adding information-reuse mechanisms when the same material is read repeatedly, or adjusting exploration budgets and termination rules when exploration runs too long.

Evaluation Without Ground-Truth Labels

HarnessPro does not rely on official test suites. Instead, a Test Generation Agent creates tests from the task description and initial execution traces. These tests are then fixed and used to evaluate all candidates. This approach raised failure recall from 41.7% to 83.3% compared to having an LLM directly judge success from logs.

Two-Stage Selection Produces Three Profiles

Discard any candidate that underperforms the initial framework on the generated tests.

Remove candidates that are Pareto-dominated (higher cost and lower success rate than another candidate).

From the remaining set, weight performance vs. cost gaps to retain three variants: Lite (cost-focused), Medium (balanced), and Heavy (performance-focused). This prevents all variants from converging on "more compute for higher success."

Benchmark Results (GPT-5.5 + Codex)

Evaluated on Terminal-Bench 2 , SWE-Bench Pro , and ProgramBench :

Heavy vs. RHO : average pass rate +2.0 percentage points; average per-task cost -26.1%.

Terminal-Bench 2: pass rate 76.3% → 80.8%; cost $1.09 → $0.72 (-33.9%).

SWE-Bench Pro (Medium): pass rate 79.0% (RHO 80.0%); cost $2.68 → $1.26 (-53.0%).

Key Improvements in Final Frameworks

Inspection of the evolved frameworks revealed three main changes:

Task-specific allocation of reasoning intensity and tool budgets.

Prompts that remind the agent to stop exploration at appropriate times.

Compression of duplicate logs and redundant instructions.

These changes directed more compute toward necessary analysis and verification while cutting repetitive consumption.

Caveat

The reported 26.1% cost reduction applies to the execution cost of the improved framework ; the compute spent on framework search and evaluation itself is accounted separately.

Bottom line: With the base model frozen, agents can repeatedly refine their own working methods to achieve both higher success rates and lower per-task costs.