Harness-of-Harness: Continuous Development & Verification for Long-Cycle AI Coding Agents

Problem: Long-Cycle Agent Drift

Existing coding agents can modify repositories, call tools, and debug local errors, but over extended tasks they accumulate requirement omissions, duplicate work, regression faults, and false acceptances. When interactions stretch to dozens of rounds, project state no longer equals the latest code: unimplemented requirements, reasons for failed fixes, and verified features scatter across early conversations or temporary logs. A new agent reading only the current repository must re-infer this context.

Another common issue: local errors are more observable than overall requirements. Agents may spend multiple rounds patching the same exception or treat "code runs" as "requirement satisfied" while functionality remains incomplete. Rapid feature addition also expands change scope, causing regressions in previously working behavior.

HoH Execution Protocol: Three Roles, Two States

HoH addresses these failure modes by structuring each iteration into three distinct roles, each invoked as an independent call of the same Harness–Model configuration but with different prompts, input contexts, and runtime permissions:

Project Planner : Selects a small, functionally complete set of tasks from the product requirements, current code, and previous evidence. The plan specifies which previously verified behaviors must be preserved.

Developer : Continues development from the existing project, with permission to modify software artifacts. It chooses implementation details and debugging tools, and performs local tests before and after changes.

QA Tester : Examines a frozen candidate version using black-box interaction tests and white-box source, configuration, and log inspection. It records evidence, gaps, and regressions for each requirement. QA cannot modify the artifact under test, so its report corresponds to a definite software version.

Between rounds HoH preserves two state types:

Artifact State (A_t) : Current source code, resources, configuration, and project metadata — what the software has implemented.

Evidence State (ℰ_t) : Test observations, verified behaviors, unmet requirements, and pending faults — which conclusions are evidence-backed.

An iteration for a given requirement S can be formalized as: Planner(S, A_{t-1}, ℰ_{t-1}) → Plan_t; Developer(Plan_t, A_{t-1}) → A_t; QA(Plan_t, A_t) → ℰ_t. Code is inherited, evidence feeds the next planning step, and together they determine how the project continues. If a behavior passed verification in the previous round, the next plan marks it as required to retain, and QA checks whether new changes caused regression.

To avoid stuffing the entire history into context, HoH persists plans, reports, and development logs to the filesystem, exposing a categorized index and reading details on demand. Role-specific tools and skills are also provided on demand; in the long-horizon case study, version records were used to trace changes and faults. Notably, the main benchmark experiments did not introduce case-specific tools, skills, or version control mechanisms, so the observed gains primarily reflect the HoH protocol itself.

Experimental Results

Benchmarks and Configurations

Three task suites were used:

GameCraft-Bench : Generate playable Godot games from natural language; 45 tasks, composite score 0–100.

FrontierSWE : Software implementation, performance optimization, and research tasks; 15 tasks.

ProgramBench : Reconstruct software from a compiled executable and documentation, evaluated by hidden behavioral tests.

Three Harness–Model combinations were tested (e.g., Codex + GPT-5.5). The main experiment compared a single vanilla development run ( Vanilla ) against three rounds of HoH ( HoH@3 ).

Quality Improvement

Across all nine (model × benchmark) comparisons, HoH@3's aggregate metric exceeded the corresponding Vanilla baseline. The average relative improvement was 52.25% . GameCraft gains appeared across mechanics, content, functional visuals, and presentation quality sub-scores.

Controlling for Compute Budget

Because three HoH rounds consume more inference budget than one Vanilla round, a Vanilla Continuation baseline was added: the original agent session simply continued for two more rounds without HoH's role separation or evidence feedback. On GameCraft-Bench with Codex + GPT-5.5:

HoH@3: 71.52 score, ~5.67M tokens (three rounds).

Vanilla Continuation (three rounds): 58.24 score, ~6.33M tokens.

HoH two rounds: 64.84 score, ~5.67M tokens.

The quality gap cannot be explained solely by more rounds or tokens, though HoH's three rounds used more tokens than Vanilla's three rounds, and the paper does not claim HoH achieves the lowest delivery cost under all tasks and budgets.

Ablation Study

Removing any one of the three cross-round mechanisms (code inheritance, evidence feedback, or role separation) reduced average quality scores. Notably, disabling code inheritance both lowered the score and increased token consumption to 11.12M, indicating that accumulating existing implementations and revising plans based on test evidence is more effective than repeatedly building from scratch. The ablation did not fully disentangle all interaction effects among the three mechanisms.

70-Round FPS Game Case Study: Fusepoint

Starting from a product requirements document and an empty project, HoH (Codex CLI + GPT-5.6-Sol, Godot engine, with dedicated engine-operation, asset-processing, UI/UX, and testing skills) developed a single-player FPS game called Fusepoint . Requirements included a ~5-minute bomb-defusal mission, sequential capture of two control points, a three-phase defusal process, 18 enemies distributed across three combat zones, and success/explosion endings. Human intervention was limited to restoring network or API availability.

The 70-round trajectory showed clear phases:

Rounds 1–27: Establish runnable project and core interactions, exposing missing requirements and defects.

Rounds 28–49: Continuous feature expansion, with repair work gradually increasing.

Later rounds: As features neared completion, more iterations focused on stability and regression handling.

By round 70 the project had logged 81 issues : 65 closed, 16 still open ; 17 issues were reopened after closure. These regressions confirm that long-horizon autonomous development still breaks previously verified functionality. HoH's value lies in linking regressions to prior passing verification records, allowing the next planner to address known failures directly instead of guessing problem history from the latest code alone.

This case demonstrates HoH's ability to maintain software state and iteration process over dozens of rounds and to produce a playable game. It remains a single-engine, single-requirement case; the project had unresolved issues at observation cutoff, and the paper does not generalize it to universal commercial software delivery capability.

Author's Assessment

The research question is well-chosen: planning drift, information loss, and regression in long tasks do not vanish simply because single-turn code generation improves. HoH translates these problems into checkable engineering constraints: one verifiable increment per round, only the Developer mutates the artifact, QA collects evidence against a frozen version, and the next round starts from both code and evidence. The design is interpretable and component contributions are localizable during reproduction.

Algorithmic novelty is limited — role-based development, incremental iteration, continuous testing, file-based memory, and version control have prior research and engineering foundations. The paper's strength is composing these methods into a fixed execution protocol and validating it with cross-model, cross-task, budget-controlled, and ablation experiments. This qualifies as systems engineering research and is worth reading for that community.

From a product implementation perspective, HoH maps to existing infrastructure: Planner defines increments and acceptance criteria, Developer modifies a Git workspace, QA combines CI, runtime checks, and interaction tests to produce evidence, and Issues plus version history track todos, regressions, and verified features. This mapping is the author's synthesis, not an empirical conclusion from large-scale enterprise deployment. Real-world adoption for cross-team collaboration, database migration, security governance, production release, etc., still requires separate design and validation.

For reproduction, the author suggests starting from Sections 3, 4.3, Appendix A, and Appendix B.5: first replicate cross-round state, frozen-candidate QA, and same-budget baseline comparisons, then consider adding domain-specific tools. For Agent Runtime researchers, these mechanisms deserve more experimental investment than simply copying the three role prompts.

Paper: Harness-of-Harness: Multi-Day Autonomous Software Development with Continual Improvement

URL: https://arxiv.org/html/2609.01481

Code example

来源：DeepHub IMBA
本文
约4000字
，建议阅读
8
分钟
给 Coding Agent 一份高层软件需求，让它在较少人工介入的情况下连续开发数天，是否能够最终得到可运行、可验收的软件？