Harness-Zero: Agent-as-Harness Distills Scaffolding into Model Weights

Harness-Zero addresses a core problem in agent deployment: the same model performs dramatically differently with or without a harness (scaffolding). Current programming agents like Claude Code and Codex derive much of their strength from external harnesses, but those optimizations remain outside the model parameters.

Core Concept: Agent-as-Harness

The paper proposes agent-as-harness : instead of hand-coded scaffolding, a harnessing agent acts as the scaffold. This harnessing agent translates guidance from an evolved reference harness K into executable corrections within the student agent's action space. The process has three steps:

Evolve and adapt: On training tasks, evolve the student's optimal harness h* and adapt it into a private reference harness K. Middleware rules become review prompts, tools become action recipes, skills and memory become diagnostic criteria and intervention guides.

Review and correct: The student proposes actions normally. Each proposed response is intercepted and reviewed by the harnessing agent using K. Reasonable actions pass (PASS); problematic ones receive a minimal, semantically complete replacement (REPLACE) written from the student's perspective and legal in the student's harness h. The review is private and does not enter the student's trajectory; the harnessing agent sees only K, not hidden answers or sandbox internals.

Distill via SFT: The reviewed trajectories are used for supervised fine-tuning. Any reviewer-perspective reasoning accidentally included is masked out.

Inference-Time Comparison

On two frontier models (GPT-5.6 Sol and DeepSeek-V4-Pro) across three benchmarks (SpreadsheetBench, AppWorld, USPTO retrosynthesis), agent-as-harness with evolved K achieves 81.1% average success , outperforming code-as-harness (78.1%) and bare mini-SWE-agent (68.6%). Notably, student and reviewer are the same model , so gains cannot come from a stronger teacher. An empty K yields only 69.2%, showing that the domain knowledge inside K drives the improvement .

Distillation Results: Shedding the Armor Makes the Model Stronger

Using Qwen3.5-9B as student and GPT-5.6 Sol as harnessing agent, after distillation the student runs with only a minimal fixed harness h (all of h*, K, and the harnessing agent removed). Macro-average success rate jumps from 23.3% to 44.3% — an absolute gain of 21.0 points, a 90.1% relative improvement . Crucially, this exceeds the base model equipped with the full evolved harness h* (41.7%) .

Control experiments show generic harnesses hurt small models: DeepAgents drags the 9B model to 20.5%, Claude Code to 15.9% — the generic tool suite is too large and noisy. Per-domain analysis reveals why distillation works better on some benchmarks: SpreadsheetBench and AppWorld rely on procedural behaviors (check state, targeted edit, verify) that review trajectories demonstrate directly; USPTO retains deep domain knowledge (reaction priors, candidate generation, molecular verification) that a single SFT pass cannot fully absorb (distilled 30.0% vs h* 38.0%).

Where Effective Supervision Comes From: Ablation on USPTO

The USPTO ablation is the most revealing. Distilling raw GPT-5.6 Sol trajectories yields no gain (12.0%). Distilling GPT-5.6 Sol with h* trajectories collapses to 3.0% — the student learns tool calls it cannot execute, exhausting step budgets. Most counter-intuitively, an oracle harnessing agent given the ground-truth answer achieves 98.6% collection success but distills to only 15.0%; meanwhile K (no answers, 59.4% collection success) distills to 30.0%. Collection success does not predict distillation value — answer access induces non-generalizable shortcut corrections, while K instills procedural behaviors the student can execute independently.

Behavioral Evidence: 82.3% Recovery of Harness-Specific Behaviors

Beyond aggregate scores, the authors detect 28 harness-specific behavior patterns (tools, middleware, memory, skills) that the base model exhibits under h* but never under minimal h. The distilled model recovers 82.3% of these behaviors on average . Examples: "check workbook structure before editing" (100%), "validate SMILES with RDKit" (100%), "protect pre-filled cells" (92%). Since the base model scores 0% on these tasks by construction, the 82.3% represents genuinely new behaviors acquired through distillation, not amplification of existing capabilities.

Harness-Zero: Harness Distillation via Agent-as-Harness https://arxiv.org/pdf/2609.24974 https://github.com/metaevo-ai/harness-zero