Harness-of-Harness: Converting Long-Running AI Agent Drift into Checkable Engineering Constraints

Why Long-Running Development Loses Control

In short tasks, we can directly review an agent's changes and decide the next step. But extending this interaction to dozens of rounds makes project state diverge from the latest code: which requirements remain unimplemented, why a fix failed, which features have passed verification — all may be scattered in early conversations or temporary logs. A new agent reading only the current repository must re-infer this information.

Another common issue: local errors are more observable than overall requirements. An agent may spend consecutive rounds patching the same exception, or treat "code runs" as "requirement satisfied" while functionality is incomplete. Rapid feature addition also expands change scope, causing regressions. HoH addresses these failure modes by making each round select a verifiable development increment, letting the developer implement and self-test, then having an independent role verify a frozen version, and finally feeding the verification results into the next planning round.

This arrangement borrows mature software engineering ideas — incremental development, continuous testing, and separation of concerns. The paper's contribution is implementing them as a repeatable agent protocol and measuring the protocol's impact on final delivery quality.

HoH Execution Protocol: Three Roles, Two States

In one HoH iteration:

Project Planner selects a set of related tasks based on product requirements, current code, and previous evidence, producing a development document. The task scope must be small enough to localize faults yet functionally complete for observable acceptance. The plan also notes previously verified behaviors that must be preserved.

Developer continues from the existing project, with permission to modify software artifacts. It chooses implementation approaches and debugging tools, and performs local tests before and after changes.

QA Tester then examines the frozen candidate version, combining black-box interaction tests with white-box source, configuration, and log inspection, recording evidence, gaps, and regressions for each requirement. QA cannot modify the artifact under acceptance, so the test report corresponds to a definite software version.

In the paper's experiments, these three roles are three independent invocations of the same Harness–Model configuration , distinguished by their prompts, input contexts, and runtime permissions. "Independent QA" means acceptance is isolated from implementation, not that a different model or human auditor is introduced. The Runtime manages role inputs, read/write permissions, and structured output validation, making responsibility boundaries traceable.

Original paper Figure 3: HoH architecture overview, including role division, development documents, software artifacts, test evidence, and tools/skills

HoH preserves two state types between rounds:

Artifact State (A_t) — current source code, resources, configuration, and project metadata, describing what the software has implemented.

Evidence State (E_t) — test observations, verified behaviors, unmet requirements, and pending faults, describing which conclusions have evidential support.

For a given requirement S, one iteration can be formalized as shown in the paper's Figure 3 (architecture overview). Code is inherited, evidence participates in the next round's decisions, and together they determine how the project continues. If a behavior passed verification in the previous round, subsequent planning treats it as a behavior to retain and requires QA to check whether new changes cause regression. Code history alone does not convey these acceptance constraints; test reports alone do not give the Developer a modifiable project.

To avoid stuffing the entire history into context, HoH persists plans, reports, and development logs to the filesystem, exposing a categorized index first and then reading specific content on demand. Role-specific tools and skills are also provided on demand; in the long-horizon case study, version records are used to trace changes and faults. Notably, the main benchmark experiments did not introduce case-specific tools, skills, or version control mechanisms, so the benchmark gains primarily reflect the HoH plan–develop–test protocol.

Experimental Results

The paper uses three task suites:

GameCraft-Bench — generate playable Godot games from natural language; 45 tasks sampled, composite score 0–100.

FrontierSWE — software implementation, performance optimization, and research tasks; 15 tasks used.

ProgramBench — reconstruct software from compiled executables and documentation, evaluated via hidden behavior tests.

The main experiment compares a single vanilla development run ( Vanilla ) against three rounds of plan–develop–test ( HoH@3 ). Across nine "model configuration × benchmark" comparisons, HoH@3's aggregate metrics exceed the corresponding Vanilla baseline; improvement magnitude varies by task and configuration. GameCraft gains appear across mechanics, content, functional visuals, and presentation quality sub-scores. This supports a scoped conclusion: within the paper's covered project construction and program reconstruction tasks, fixing the underlying model and improving the execution protocol improves final software artifact quality.

Original paper Table 1: FrontierSWE's Dominance is a pairwise relative advantage metric over the experimental configuration pool, not a task completion rate

However, three-round HoH consumes more inference budget than single-round Vanilla. To rule out "more runs = better", the authors add Vanilla Continuation : continue the original agent session for second and third rounds without HoH's role separation and evidence feedback. For Codex + GPT-5.5 on GameCraft-Bench:

HoH@3: 71.52 score, ~5.67M tokens (two rounds: 64.84 score)

Vanilla Continuation (three rounds): 58.24 score, ~6.33M tokens

Same three development rounds, HoH outperforms Vanilla Continuation with fewer tokens, indicating the quality gap isn't fully explained by round count or token budget. However, HoH's three rounds used more tokens than two-round HoH, and the paper does not prove it achieves the lowest delivery cost under all tasks and budgets.

Ablation experiments further examine cross-round mechanisms (Codex + GPT-5.5, GameCraft-Bench, three rounds):

Removing any mechanism reduces average quality score.

Removing code inheritance especially lowers score and increases token consumption to 11.12M tokens.

This suggests that accumulating existing implementations and revising plans based on test evidence is more effective than repeatedly building from scratch, at least for this task set. The three mechanisms interact; the ablation does not fully disentangle all interaction effects.

70-Round FPS Game: Long-Horizon Capability and Persistent Regressions

The authors also tasked HoH with building a single-player FPS game Fusepoint from a product requirements document and an empty project. Requirements included a ~5-minute bomb-defusal mission, sequential capture of two control points, a three-phase defusal process, 18 enemies distributed across three combat zones, and success/explosion endings. The system used Codex CLI + GPT-5.6-Sol and Godot, equipped with engine operation, asset processing, UI/UX, and testing skills. Human intervention was limited to restoring network or API availability, not planning, coding, debugging, or acceptance.

The 70-round development log shows clear phases:

Rounds 1–27: establish runnable project and core interactions, while exposing missing requirements and defects.

Rounds 28–49: continuous feature expansion, with repair work gradually increasing.

Later rounds: as features near completeness, more iterations focus on stability and regression handling.

Original paper Section 5.2 Fusepoint development trajectory: horizontal axis is iteration rounds, blue curve shows unresolved issue count trend, bottom annotations show newly opened and closed issues

By round 70, the project recorded 81 issues total, with 65 closed and 16 still open ; 17 issues were reopened after closure . These regressions demonstrate that long-horizon autonomous development still breaks previously verified functionality. HoH's value is linking regressions to prior passing verification records, enabling the next planner to address known failures directly instead of inferring problem history solely from the latest code.

This case study shows HoH can sustain software state and iteration process for dozens of rounds and produces a playable game. It remains a single-engine, single-requirement-set case; the project had unresolved issues at observation cutoff, and the paper does not generalize it to universal commercial software delivery capability.

Summary

The research question is well-chosen: planning drift, information loss, and regression in long-running coding agents do not disappear automatically as single-turn code generation improves. HoH converts these problems into checkable engineering constraints: one round handles one verifiable increment; only the Developer modifies artifacts; QA collects evidence against a frozen version; the next round starts from existing code and evidence. The design is not overly complex, and reproduction can clearly isolate each component's effect.

Algorithmic novelty is relatively limited. Role-based development, incremental iteration, continuous testing, file-based memory, and version control all have prior research or engineering foundations. The paper's strength lies in composing these methods into a fixed execution protocol and validating it with cross-model, cross-task, budget-controlled, and ablation experiments. By that standard, this is a systems engineering research contribution worth reading.

From a product implementation perspective, HoH maps to existing engineering infrastructure: Planner defines increments and acceptance criteria; Developer modifies a Git workspace; Tester combines CI, runtime checks, and interaction tests to generate evidence; Issues and version history track todos, regressions, and verified features. This is the author's synthesis, not an empirical conclusion from large-scale enterprise R&D deployment. Real-world scenarios like cross-team collaboration, database migration, security governance, and production release still require separate design and validation.

For reproduction, start from Section 3, Section 4.3, Appendix A, and Appendix B.5 : first replicate cross-round state, frozen-candidate QA, and same-budget baseline comparison, then consider adding domain-specific tools. For Agent Runtime researchers, these mechanisms are more worth experimental investment than simply copying three role prompts.

Paper: Harness-of-Harness: Multi-Day Autonomous Software Development with Continual Improvement — https://arxiv.org/html/2609.01481

Code example

如果你希望把模型、工具、状态、权限、执行环境和插件生命周期放在同一张架构图里理解，可以继续阅读：