Skip to content
Doc2Harness: Benchmarking LLMs on Long-Horizon Harness Engineering – GPT-6 Astra Tops at 53% Pass Rate

Doc2Harness: Benchmarking LLMs on Long-Horizon Harness Engineering – GPT-6 Astra Tops at 53% Pass Rate

Researchers from CAS, Tsinghua, PKU, and Southeast University introduce Doc2Harness, the first deterministic benchmark evaluating LLMs' ability to build coding agent harnesses from a contract skeleton and dev doc; 19 models tested, GPT-6 Astra achieves highest pass rate of 53.05% on 328 test cases, revealing significant room for improvement in long-horizon engineering.

Machine Learning Algorithms & Natural Language Processing
Machine Learning Algorithms & Natural Language Processing
Machine Learning Algorithms & Natural Language Processing

Motivation: Why Harness Engineering Needs a Dedicated Benchmark

Building a harness is a software engineering task, but existing benchmarks (SWE-bench Pro, NL2Repo-Bench, Commit0) use mature open-source repositories that likely appeared in training data. Harnesses are recent, fast-evolving software, so models cannot rely on memorization. Moreover, existing benchmarks rely on unit tests tied to fixed interfaces, whereas a harness can have multiple valid implementations. Since a harness does not complete downstream tasks itself, its correctness can only be judged by observing its external behavior when driven by a model.

Overall Design: Build in One Container, Evaluate in Another

Doc2Harness splits evaluation into two isolated phases to reduce confounding factors. An animated diagram illustrates the flow.

Phase 1: Build

The tested agent consists of a build model and a fixed builder harness (OpenCode, chosen because it is open-source and avoids vendor bias). The agent reads the development document and starts from a contract skeleton, writing a candidate harness. Each commit triggers a contract check verifying that the deliverable compiles, required commands are present, no placeholder code remains, and the three contract interfaces (CLI, Headless entry, automation API) are in place. Contract checks are purely syntactic; they do not run test cases. If checks pass, the deliverable is frozen as a candidate harness. If not, the agent may revise and resubmit, up to 3 submissions within a 2-hour wall-clock limit.

Phase 2: Evaluate

The frozen candidate harness moves to an isolated container. Each test case runs in a fresh container with a fresh set of initial files. A mock model follows a preset script to drive the harness through a complete session. Using a mock model eliminates randomness from real LLM outputs, making every judgment deterministic. Four observable channels are recorded: requests sent to the model, files on disk, process exit codes, and command outputs. Some test cases have positive controls (see below). A test case passes only if it passes itself and all its associated positive controls also pass. Positive controls do not count toward the pass rate. The benchmark ranks build models by their candidate harnesses' overall pass rate across all test cases.

Task Inputs: Development Document and Contract Skeleton

Both inputs are designed to minimize reliance on the build model's prior knowledge, forcing it to follow the specification.

Development Document: Integrating 12 Frontier Harnesses

The document was created in three steps:

Collect capability list. The team scraped user documentation from 12 frontier harnesses (Claude Code, Kimi Code, Gemini CLI, Qwen Code, OpenCode, OpenClaw, Codex CLI, Aider, OpenHands, Cline, Crush, Cursor), extracted candidate capabilities, supplemented each with invocation forms, parameter semantics, and boundary behaviors, then merged, deduplicated, and trimmed to form a capability list.

Write clauses. Clauses are the atomic units of the document. The document has 10 chapters covering deliverables & runtime interfaces, built-in tools, top-level commands & subcommands, permissions & role modes, global parameters, in-session commands & shortcuts, persistent memory, convention files, consistency rules, and exit codes. Each clause specifies syntax, meaning, and expected behavior observable through one of the four channels; beyond the three contract interfaces, clauses impose no implementation constraints.

Eliminate ambiguities. Clauses from 12 sources inevitably conflict. Semantically overlapping commands were merged (e.g., context compression called /compact in Claude Code and /summarize in Cursor, unified as /compact). Same-name but different-meaning concepts were split (e.g., "plan" as a read-only permission mode in Claude Code vs. separate architect/editor roles in Aider). The document places read-only constraints under permissions and proposal/execution separation under roles, each as independent clauses.

Contract Skeleton: ~70 Lines, Rest Written by Model

The skeleton is distilled from mini-swe-agent: 5 Python source files totaling ~70 lines, plus an install script, an entry script, and a project config, containing no functional implementation. Three interfaces are immutable external contracts: CLI, Headless entry, and the automation API. The thinner the skeleton, the more the model must write. A representative candidate harness has ~30 Python files and ~16,000 lines, ~99.6% written by the build model. Each build starts from a fresh empty container with no internet access except a designated model gateway. A pre-check ensures no extra files exist and OpenCode starts cleanly; any violation aborts the build.

Test Case Synthesis: Three Methods Used Alternately

Each test case must anchor to a specific clause in the development document and discriminate between implementations. Three synthesis methods alternate: the first two broaden coverage, the third deepens scrutiny; when marginal yield drops, the team switches methods.

4.1 Public Issue Mining

Based on the hypothesis that defects in mature harnesses (trained on human code) reflect errors models will make. The team pulled all issues from four frontier harness repos, sorted by relevance to the dev doc, and clustered reports describing the same phenomenon into candidate test cases. Candidates anchoring the same clause with overly similar observed effects were deduplicated, keeping the one with more complete evidence. Candidates must be expressible as deterministic assertions and not covered by existing test suites.

4.2 Cross-Harness Comparison

Public issues are limited in quantity and quality. This method starts from source code. The team partitioned a candidate harness's source into modules per the dev doc's features, did the same for mature harnesses (Kimi Code, Qwen Code, etc.), and compared module by module: which features mature implementations handle that the candidate misses, and where behaviors differ on boundaries. Differences are not necessarily defects; each must be traced back to a clause's intent in the dev doc to become a candidate test case.

4.3 Single-Variable Extension

From a seed test case proven to discriminate, create variants by changing one variable while keeping the assertion fixed. Four variable categories: tools (same goal achievable via different tools; candidate may implement only a subset), input forms (e.g., path traversal test using symlinks to files, directories, multi-level symlinks, relative-path symlinks), erroneous parameters (same coding error on different parameters, e.g., treating numeric 0 as default), result locations (same behavior's output may appear in stdout, stderr, or error messages). New candidates undergo similarity checks against the seed; too-similar ones are discarded. Remaining become test cases and serve as seeds for the next round. Clauses without any seed test case are covered by the first two methods, necessitating alternation.

Positive Controls: Excluding Vacuous Passes

5.1 The Vacuous Pass Problem

A passing test case does not guarantee the harness implemented the requirement. Example: path traversal test – mock model asks harness to read a symlink pointing outside workspace; assertion checks that external file content does not appear in the request to the model. Content absence could mean the harness intercepted the out-of-bounds read, or that it cannot read any file at all. Both satisfy the assertion, but only the former meets the requirement; the latter is a vacuous pass.

5.2 Three Types of Positive Controls

Positive controls are designed to exclude vacuous passes. They share the same scenario and initial files as the guarded test case, but change the input or access path to a situation not constrained by the tested clause, and assert a result that must appear if the observation premise holds. Assuming the assertion itself is correct, vacuous passes have only three causes, each addressed by a control type:

Channel Control: Verifies tool results can reach the observed channel. Example: read a regular file, confirm its content appears in the request. If this fails, the harness doesn't write tool results to this channel, so absence of forbidden content proves nothing about interception.

Same-Form Control: Excludes failures that would happen even without the tested constraint. Input form unchanged, only access path changed – e.g., use bash tool to access the same symlink, confirm it resolves outside workspace and file is readable. The path-traversal clause constrains file tools; bash commands are out of scope. If this fails, the harness cannot complete even unconstrained access, so its failure on the tested access is unrelated to interception.

Reverse Control: Excludes blanket rejection of a class of inputs. A harness rejecting all symlinks would pass the test case but also reject symlinks pointing inside the workspace, which are legitimate in normal use. Replace input with an internal-workspace symlink, perform same read, confirm content still appears. If this fails, the earlier pass was not due to recognizing out-of-bounds access.

A figure shows typical implementations' results on the test case and three controls. Three implementations pass the test case but each fails one control type, all ultimately recorded as FAIL. Test cases sharing an observation premise share a positive control; cases with multiple premises are guarded by multiple controls. If any positive control fails, all test cases it guards are marked FAIL regardless of their own results.

Test Suite Statistics

Based on the dev doc and three external contracts, 328 test cases fall into 9 disjoint categories (left chart): Session State 84, File I/O 75, Permissions 61, Headless 40, Direct Drive 33, Interactive Terminal 20, Deliverable Check 7, Startup 7, Extension Discovery 1. Right chart shows proportion of test cases guarded by positive controls per category. 130 test cases have at least one positive control: 114 with one, 15 with two, 1 with three. File I/O and Permissions have highest coverage; Interactive Terminal lowest; Deliverable Check marked NA because contract checks already fully verify those items. The most widely used positive control guards 6 test cases, confirming that symlinks pointing outside workspace are indeed readable.

Experimental Results

19 frontier models from 11 vendors evaluated, each building independently three times; all three candidate harnesses scored, totaling 57 builds.

7.1 Far from Saturation

As shown in the lead figure, GPT-6 Astra leads with 53.05% pass rate; last-place Hy3 scores 27.44%, a ~26 percentage-point gap. Even the strongest model fails nearly half the test cases. An animated GIF and a bar chart illustrate per-model performance. The gap between top two models appears mainly on boundary scenarios. On normal scenarios, Opus 5 leads at 80.68% vs. GPT-6 Astra's 72.46%; on boundary scenarios, GPT-6 Astra leads at 48.97% vs. Opus 5's 37.02%. Categories differ in discriminative power: Permissions average pass rate 43.7%, strongest-weakest gap 50.8 points, 51 of 61 test cases discriminate models. File I/O average similar at 39.6% but gap only 20.9 points. Interactive Terminal averages 79.9%, near saturation. 56 test cases were not passed by any candidate harness.

7.2 Rankings Vary Widely Across Categories

A heatmap shows per-category rankings. Same model's rank varies up to 16 places across six major categories: Opus 4.6 ranks 3rd on Direct Drive but 19th on Permissions; Grok 4.7 ranks 1st on File I/O but 11th on Session State. However, this does not imply trade-offs between categories. All 15 pairwise correlations between categories are positive, averaging +0.521, indicating model differences reflect overall capability rather than category-specific strengths. KR-20 coefficient across all test cases is 0.913, showing high internal consistency of the test suite.

7.3 Finer Module Granularity Correlates with Lower Scores

The team analyzed source code of all 57 candidate harnesses. A scatter plot shows each model's three builds (left: average per model; right: distribution per model). Builds from the same model have similar size; across models size varies greatly: 95.9% of line-count variance is between models. Line count correlates with pass rate at 0.614. Controlling for line count, module count correlates negatively with pass rate (partial correlation -0.423, robust across leave-one-model-out: -0.484 to -0.309). Lines per module correlates positively with pass rate (partial correlation +0.504). At equal code volume, candidates that scatter cross-cutting functionality across more modules score lower.

Conclusion

Doc2Harness turns "can a model build a harness?" into a deterministically measurable question. The build model, inside a fixed builder harness, follows a dev doc integrating 12 frontier harnesses' features, starting from a ~70-line contract skeleton, to produce a complete candidate harness. A mock model then drives it through 328 test cases, observing four channels, with positive controls systematically excluding vacuous passes. Build and evaluation are isolated; judgment relies solely on externally observable behavior; every pass must survive positive-control validation. This addresses the pervasive vacuous-pass risk in assertion-based evaluation. Results show this capability is far from saturated: the best model fails nearly half the test cases, with failures concentrated on implementation choices about how cross-cutting functionality is partitioned across modules. The authors hope Doc2Harness becomes a standard benchmark for long-horizon construction ability.

Original Source

Signed-in readers can open the original source through BestHub's protected redirect.

Sign in to view source
Republication Notice

This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactadmin@besthub.devand we will review it promptly.

coding agentsharness engineeringLLM benchmarkrecursive self-improvementdeterministic evaluationDoc2Harnesslong-horizon code generationpositive controls
Reader feedback

How this landed with the community

Sign in to like

Rate this article

Was this worth your time?

Sign in to rate
Discussion

0 Comments

Thoughtful readers leave field notes, pushback, and hard-won operational detail here.