Google's Harness²: Agents Self-Improve Their Own Harnesses Without Weight Updates

Introduction: The Harness Problem

Modern AI agents consist of a model plus a Harness —the prompt, memory, skills, tool specifications, and sub-agent orchestration. The same model can perform drastically differently across harnesses, yet harness tuning is manual, costly, and static. Recursive self-improvement (RSI) sounds promising but stumbles in open-world tasks because there are no ground-truth answers or reliable verifiers, and improvements made for one task rarely transfer to another.

Two Barriers in Open-World Tasks

The paper identifies two core difficulties: (1) No standard answers —tasks are unseen at design time and novel at runtime, so there is no ground truth or verifier to judge whether a harness change actually helps. (2) Experience cannot be reused —a harness crafted for task A often fails on task B, and blindly inheriting it introduces noise when task distributions shift. The key insight is that the improvement process itself —how defects were diagnosed, which edits were tried, what execution evidence was observed—is valuable experience that should be preserved.

Harness²: Dual-Layer Recursion

The framework operates at two levels:

1. Task-Level Recursion: Propose–Probe–Compose

This is the engine for a single task, controlled by a budget parameter k:

Propose : Run the current harness to get a reference execution. Generate two candidate harnesses—a general candidate targeting cross-domain defects (planning, tool use, context management, delegation) and a domain candidate targeting domain-specific knowledge, processes, and constraints.

Probe : Execute each candidate on the same task, model, and environment. Compare its behavior against the reference execution to produce a behavioral comparison showing whether the target defect was mitigated, unchanged, or new issues appeared.

Compose : Synthesize all evidence to selectively merge, revise, adopt, or reject candidates. Every decision must be backed by a concrete observation, not intuition.

Critical design point : The behavioral comparisons provide evidence for synthesis; they do not verify correctness because execution is stochastic and models share biases.

2. Two Expansion Modes: Parallel vs. Serial

With budget k, parallel recursion expands all candidates at once and runs multiple independent synthesis sessions; serial recursion feeds the previous step's harness and evidence into the next step, allowing later steps to correct earlier edits. When k=1 they are equivalent.

3. Harness-Level Recursion: The Improvement Library

After each task, the system reviews the full improvement trace and stores adopted harnesses plus evidence-backed lessons in an improvement library . Each entry contains an executable harness and an EXPERIENCE.md documenting applicable scenarios, lessons, and evidence. Two safeguards apply: (a) the next task still starts from a unified initial harness; library components are only selectively reused or adapted during Propose, never auto-inherited. (b) If evidence is insufficient, the reflection step refuses to write a new entry—mere selection of an output does not constitute transferable evidence.

The harness has 8 editable components (system prompt, guardrails, memory, skills, tool specs, scripts, sub-agents, middleware plugins), each mounted as a file at runtime; edits are file modifications.

Experimental Results

Benchmarks: WorkBuddy-Bench (code, web, office), JobBench (end-to-end work tasks), LAB (legal work, Partial metric). Models: Gemini 3.5 Flash and GLM-5 . Base harnesses: OpenCode and Codex (4 configurations). Default harness editor: Claude Opus 4.8 .

Single-step improvement (k=1) vs. base harness:

GLM-5 + OpenCode: WorkBuddy 64.2 → 75.7 ( +11.5 ), JobBench 20.6 → 24.2 (+3.6), LAB Partial 59.6 → 71.4 ( +11.8 )

GLM-5 + Codex: WorkBuddy 64.4 → 75.3 (+10.9), JobBench 23.6 → 24.8 (+1.2), LAB Partial 58.1 → 65.9 (+7.8)

Gemini 3.5 Flash + OpenCode: WorkBuddy 72.4 → 77.5 (+5.1), JobBench 31.5 → 36.0 ( +4.5 ), LAB Partial 79.9 → 81.9 (+2.0)

Gemini 3.5 Flash + Codex: WorkBuddy 74.5 → 77.5 (+3.0), JobBench 33.9 → 35.3 (+1.4), LAB Partial 71.6 → 75.6 (+4.0)

Key conclusions:

All four configurations improve; max gains: WorkBuddy +11.5, JobBench +4.5, LAB +11.8 points.

Weaker models benefit more—GLM-5 after one step surpasses Gemini 3.5 Flash's base scores on WorkBuddy (75.7 vs 72.4, 75.3 vs 74.5).

Baselines are unstable: ReasoningBank drops up to 6.6 points on code; Harness Scaling falls below base in 7 of 12 benchmark-config pairs. Harness² beats all baselines on WorkBuddy avg and JobBench across all 4 configs.

More budget helps: parallel k=5 adds WorkBuddy +2.9–3.9, JobBench +3.0–4.2, LAB +2.6–6.7 points over k=1. For Gemini+OpenCode, JobBench gain rises from 4.5 to 7.5, WorkBuddy from 5.1 to 8.4.

Harness² uses fewer or similar execution steps vs. baselines; its compute-performance trade-off lies on the empirical Pareto frontier.

Ablation Study (Gemini 3.5 Flash + OpenCode, k=1)

Full Harness²: WorkBuddy 77.5, JobBench 36.0

Without improvement library: WorkBuddy 74.4, JobBench 34.2

Without scope division (merged editor): WorkBuddy 74.6, JobBench 32.3

Without composition (single candidate): WorkBuddy 75.9, JobBench 34.1

All three designs contribute; scope division has the largest impact on JobBench (-3.7).

Direct editing fails without the library. Letting the editor modify the base harness without reference execution, probes, or library degrades performance (Code 73.4→63.2, Web 69.9→65.0). A library built with k=1 restores base performance; k=5 library yields Code +0.9, Web +4.4, Office +5.1. Deeper exploration produces more valuable experience.

Experience transfers across benchmarks. A library accumulated on JobBench improves WorkBuddy single-step by +0.6, +0.5, +2.9 on its three domains (retrospective test, not online adaptation).

Editor need not be stronger. Same-model self-improvement works for five models (GLM-5 through Claude Opus 5); Opus 5 gains +3.3/+5.5/+3.0 on the three benchmarks, GLM-5 gains +13.7 on LAB.

Case Study: JobBench Task (Gemini 3.5 Flash + OpenCode)

Base harness score: 43.1. Failure traced to process defects : multiple source files not cross-checked, output file not re-opened for verification. Harness² applied two targeted edits:

General intervention : Added process-level guardrails requiring inventory of all inputs, tracing every number to its source, and a verification script to re-open and check the output file.

Domain intervention : Added a dedicated reconciliation skill covering schedule, venue, labor, budget files, plus tool specs for numeric format handling.

All four edits were retained after comparison with the reference execution. Re-run score: 77.8.

Running the Repository

Apache-2.0 license. Requirements: Linux, Python 3.12, uv. No model credentials needed for local demo:

uv venv --python 3.12 .venv
source .venv/bin/activate
uv pip install -e '.[render]'
harness2-demo --output runs/demo --k 2

Demo uses a deterministic editor and real Python execution, showing both recursion modes. For real benchmarks (e.g., LAB) you need Node.js 20+, Google Cloud CLI, and a Vertex AI project:

harness2-setup lab
harness2 --bench lab --k 1 --mode parallel --only-domains antitrust-competition --max-tasks 1
harness2-judge --bench lab --k 1 --mode parallel
harness2-report --bench lab --k 1 --mode parallel

Flags: --k (task-level budget), --mode (parallel/serial), --substrate cx (switch to Codex executor), --dry-run (preview task selection). Results saved under runs/ (artifacts, adopted harnesses, reflection notes, eval results); same command resumes from checkpoint.

Limitations & Caveats

Comparison ≠ verification. Behavioral comparisons are limited by execution randomness and shared model biases; they are evidence, not correctness proofs.

Editor-executor strength gap. Main experiments use stronger Opus 4.8 as editor; same-model experiments mitigate but absolute gains aren't directly comparable.

Cost. Improvement consumes tokens; the paper's compute analysis includes both execution and improvement costs and shows Harness² on the Pareto frontier, but per-use economics must be evaluated individually.

Not universally best. On LAB Partial, Gemini 3.5 Flash + Codex is beaten by Meta-Harness; on some sub-tasks offline-optimized Meta-Harness scores higher.

Evaluation security. Executor network access is unrestricted by default; Codex sandbox is off; public benchmark answers/scoring rubrics may be online. Production evaluation requires manual network restrictions.

Not an official Google product. Repository disclaimer states it is not Google-supported.

Conclusion

Harness² reframes harness engineering as a learnable object: keep model weights fixed, treat how to improve the harness as the optimization target. It replaces ground truth with same-task behavioral comparison, and replaces blind inheritance with an evidence-conditioned improvement library. As model capabilities converge, harness engineering may become the next differentiator; whether this self-modifying mechanism reliably generalizes across real-world workloads remains for the community to validate.

Sources

Paper: Harness²: Recursive Agent Harnessing for an Open World , arXiv:2610.14224 – https://arxiv.org/html/2610.14224v1

Code: https://github.com/google-research/harness2

Project page: https://harness2.github.io