MemoHarness: Agent Evolution Shifts from Model Weights to External Harness Systems

The article analyzes the MemoHarness paper (arXiv:2607.14159), which proposes shifting agent evolution from model-internal parameter updates to an external, versionable control layer called the Agent Harness.

Six-Dimensional Harness Space

MemoHarness decomposes the harness into six editable control planes:

D1 Context Assembly (pre-call input construction): organizes instructions, constraints, retrieved materials, and examples; supports structured prompting, few-shot examples, or context compression.

D2 Tool Interaction (external tools & retrieval): decides when to call tools, how much to retrieve, and whether to rerank evidence.

D3 Generation Control (decoding & budget): adjusts max output tokens, temperature, and candidate sampling.

D4 Task Orchestration (workflow topology): chooses between single-call and multi-stage flows such as plan–execute–refine.

D5 Memory Management (cross-call state): retains key state, summarizes trajectories, and evicts stale context.

D6 Output Processing (post-call handling): extracts answers, validates formats, and selects fallback strategies on failure.

This decomposition improves diagnosability: a failure can be traced to a specific dimension (e.g., retrieval flooding, wrong tool order, format mismatch) rather than being blamed on the model.

Two-Phase Learning Process

Development / search phase (called “training-time optimization” in the paper, but model weights stay frozen):

Starts from a minimal harness: no examples, no structured scaffolding, no external tools, deterministic single call, no cross-call memory, raw output returned.

Runs different harness configurations on development tasks with reference answers.

Selection rule: correctness first—task score ranks candidates; token cost breaks ties only. This avoids converging to “cheap but wrong” configs.

Each execution saves a case-level experience: task features, harness config, diff from previous config, full model/tool trajectory, score, token usage, and diagnosed failure dimensions.

Periodically, global patterns are induced across cases (which strategies work, which fail, cross-dimension interactions). Case experiences + global patterns form a dual-layer experience base.

Test phase (experience base frozen):

For each new unlabeled task, the system retrieves similar successful cases, failed cases, and global patterns.

Adapts the global harness into a task-specific configuration in a single shot—no labels, no feedback loop, no multi-round search.

The article emphasizes that MemoHarness’s “learning” occurs entirely in the external system: the model stays fixed while context, tools, orchestration, memory, and output handling change based on past execution experience.

Experimental Results

Evaluated on three task suites:

Terminal-Bench (long-horizon terminal operation): 0.722 → 0.806

LiveCodeBench (single-turn code generation): 0.900 → 0.967 (near saturation, smaller gain)

FinanceAgent (multi-step tool use & reasoning): 0.600 → 0.767 (larger gain, suggesting harness optimization helps more on tool-intensive, long-horizon tasks)

Cross-dataset transfer (Terminal-Bench harness applied to other benchmarks): improvements on MMMLU, StrongReject, SWE-Bench Pro; no change on HumanEvalFix and Reasoning-Gym-Easy; mixed on LawBench. Conclusion: experience has selective transferability, not universal optimality.

Cross-model transfer (harness searched with GPT-5.3-Codex applied to six other models): average success-rate increase +0.098 (range +0.038 to +0.233). Suggests some strategies are model-agnostic, but not proof of full model independence.

Cost : Terminal-Bench 18 held-out tasks used 14.18M input tokens (13.32M cached). At public pricing, $6.89—lower than Codex and Claude Code in the same table, higher than OpenCode. Paper notes this advantage depends on high cache reuse; without it, cost structure may differ.

Limitations Acknowledged by Authors

Terminal-Bench main eval uses only 18 held-out tasks; point estimates without confidence intervals or significance tests.

Baselines use different models, tools, or product interfaces—not strict same-model harness ablations.

Experience base, global patterns, and case-based adaptation not ablated individually on all tasks.

No online accumulation during deployment: each test task gets one case-based adaptation; results are not written back to drive long-term updates. “Cross-deployment online experience accumulation” listed as future work.

A contemporaneous paper (“Rethinking the Evaluation of Harness Evolution for Agents”, arXiv:2607.12227) raises a broader evaluation concern: automatic harness evolution repeatedly uses task feedback and inference budget, so it must be compared against budget-matched parallel sampling, sequential correction, etc., with strict separation of search tasks and final eval tasks; otherwise gains may stem from more attempts or benchmark overfitting.

Implications for Engineering

Harness changes are explicit, versionable, testable, and rollbackable—unlike opaque model-weight updates. Enterprises can accumulate proprietary execution assets (which tasks need stronger retrieval, which failures come from overlong context, which tool combos error-prone, which outputs require human review) in experience bases, eval suites, tool interfaces, and orchestration rules, reducing dependence on the base model. The model becomes a swappable reasoning engine; the hard-to-replicate capability moves to the external “operating system.”

Agent’s true evolution may not be the model. At minimum, the model is no longer the only evolvable part.

References: [1] Yue Huang et al. “MemoHarness: Agent Harnesses That Learn from Experience.” arXiv:2607.14159, 2026-07-14. [2] MemoHarness official repo: https://github.com/HowieHwong/MemoHarness. [3] Yike Wang et al. “Rethinking the Evaluation of Harness Evolution for Agents.” arXiv:2607.12227, 2026-07-14.