MemoHarness: Agent Evolution Moves Outside the Model to External Harness Systems

01 Agent Failures May Not Stem from the Model

Evaluating an agent often starts with asking which model it uses. However, in complex tasks like code development, terminal operations, and data analysis, the base model is only one component. The same model placed in different tool environments, context strategies, and workflows can yield completely different outcomes.

The paper terms these external mechanisms surrounding the model Agent Harness . It is not merely a prompt but the complete control layer that transforms a base model into an executable agent: what information the system shows the model, which tools it may call, how many model-invocation rounds a task requires, which states must be retained, how the generation budget is allocated, and how the final answer is extracted and verified.

Common practice configures a single global harness for all tasks. The problem is that task differences are far larger than the model interface suggests: a simple coding problem may finish in one call, while a long-horizon terminal task needs continuous environment reading, state saving, exception handling, and result verification. A fixed harness either burdens simple tasks with unnecessary orchestration overhead or leaves complex tasks lacking sufficient retrieval, memory, and verification.

MemoHarness addresses this mismatch: when an agent fails, we should not only blame "the model isn't smart enough" but also judge whether the external control strategy truly fits the current task.

02 It Optimizes the Entire Runtime, Not Just a Prompt

Recent years have seen multiple agent auto-optimization routes: automatic prompt rewriting, workflow search, distilling successful experience into skills or memory. They typically touch only one local part of the system. MemoHarness differs by treating the entire harness as a learnable, modifiable object.

This decomposition first solves diagnosability. A task failure could stem from excessive retrieval drowning key signals, illogical tool-call order, or correct model conclusions rejected due to format mismatches. Different errors map to different fix locations. Attributing all issues to prompts merely stacks instructions without confirming which component actually helps.

The six control dimensions make every change explicitly recorded. For example, the system can adjust only retrieval top-k, add a single result verifier, or switch from a single call to a "plan–execute–correct" multi-stage flow. Thus the harness becomes a set of comparable, testable, and rollbackable engineering configurations rather than an opaque whole.

This is the key difference between Harness Engineering and Prompt Engineering: the latter focuses on what instructions the model should receive; the former cares about the system in which the model operates.

03 How MemoHarness Turns Execution Trajectories into Experience

The paper divides the process into two phases. Notably, "training-time optimization" does not train model weights; it searches for better harnesses on development tasks with reference answers. The base model stays frozen; all system changes occur in the external control layer.

Search starts from a minimal initial harness: no examples, no structured scaffolding, no external tools, deterministic single call, no cross-call memory, raw output returned directly. The system runs different harness configurations and scores them by task results. The selection rule is "correctness first": task score ranks candidate harnesses; only when scores tie does token consumption serve as a secondary criterion. This avoids converging to "cheap but wrong" configurations.

Each execution saves more than a score. MemoHarness records task features, current harness, delta from previous configuration, full model and tool trajectories, result score, token usage, and diagnostics of primary failure dimensions. These form case-level experiences. The system also periodically extracts global patterns from multiple cases, inducing which strategies work, which fail easily, and how the six dimensions interact. Together they constitute a two-layer experience base.

Here, Memory is no longer traditional conversation history. It remembers what runtime strategies the system has tried, on which tasks they succeeded, and why they failed.

In the test phase the experience base is frozen. Facing a new task without reference answers, the system retrieves similar successful cases, failure cases, and global rules based on the task description and features, then adapts the searched global harness into a task-specific configuration. This phase uses no labels, accepts no task feedback, and does not restart multi-round search. It performs a one-shot adaptation based on historical experience, not unlimited self-modification during execution.

04 Experiments Show Gains, But Not a Universal Harness

Evaluation covers three task categories: long-horizon terminal operations (Terminal-Bench), single-turn code generation (LiveCodeBench), and multi-step tool-use financial analysis (FinanceAgent). Reported final validation-set results: Terminal-Bench 0.722 → 0.806, LiveCodeBench 0.900 → 0.967, FinanceAgent 0.600 → 0.767.

Improvement is uneven. LiveCodeBench baseline nears saturation, leaving little room; FinanceAgent requires multi-step reasoning, tool use, and state management, so later search still finds effective changes. The paper suggests harness optimization may be more valuable on long-horizon, tool-intensive tasks—a plausible claim needing larger-scale confirmation.

Cross-dataset results show "selective transfer." Harnesses learned from Terminal-Bench improve MMMLU, StrongReject, and SWE-Bench Pro, but HumanEvalFix and Reasoning-Gym-Easy remain unchanged, and LawBench is inconsistent. No single universally superior template emerges; rather, some control experiences can be reused across tasks.

Cross-model experiments apply harnesses searched with GPT-5.3-Codex to six other models, yielding an average success-rate increase of 0.098 under the paper's setup. All tested models improve, but magnitudes range from 0.038 to 0.233. This hints that some strategies may be more than model-specific prompt tricks, yet it does not prove full model independence.

Cost results depend on assumptions. On 18 held-out Terminal-Bench tasks MemoHarness used 14.18M input tokens, of which 13.32M were cached. At public pricing, total cost is $6.89—lower than Codex and Claude Code in the same table but higher than OpenCode. The paper explicitly notes this advantage relies on high cache reuse; if deployment cannot cache experience contexts, cost structure may differ markedly.

05 The Real Shift: Evolution Moves from Model Internals to External Systems

MemoHarness's most important cognitive contribution is not another auto-optimization algorithm but redrawing the boundary of "capability evolution." Previously, new knowledge and strategies were written into model weights; in the harness route, improvements manifest as adding a tool, changing retrieval strategy, compressing context, adjusting workflow, or adding new output verification rules.

This shift matters for engineering deployment. A capability change inside model parameters is often unexplainable, whereas every harness modification can be versioned and compared on fixed benchmarks for success rate, cost, latency, and error types. Regressions can be rolled back; risky tools or memory strategies can be disabled individually.

Further, enterprise agent capability may no longer depend solely on the underlying model. As executions accumulate, organizations accumulate proprietary experience assets: which tasks need stronger retrieval, which failures stem from overlong context, which tool combinations are error-prone, which results require human approval. These live in experience bases, eval sets, tool interfaces, and orchestration rules—not in model weights.

Thus future inter-enterprise agent gaps may not come only from who accesses a stronger model. Even with the same base model, different organizations can develop vastly different system capabilities due to distinct execution trajectories, error diagnostics, evaluation systems, and harness versions. The model becomes a replaceable reasoning engine; the hard-to-replicate part gradually moves outside the model.

06 How Far from True Long-Term Autonomous Evolution?

Describing MemoHarness as "already achieving self-evolution" overstates it. First, development-phase harness search relies on tasks with reference answers; correctness signals come from external scoring, not the system's own stable judgment of improvement. The test phase uses no labels but depends on a previously accumulated and frozen experience base.

Second, the current system does not continuously absorb new experience during deployment. Each test task receives a one-shot case-based adaptation, but execution results are not automatically written back to drive the next long-term update. The paper lists "cross-deployment online experience accumulation" as future work. It is closer to an "adaptive harness that learns from development experience" than to continuous autonomous evolution in open environments.

Experimental boundaries are also clear. Terminal-Bench's main evaluation uses 18 held-out tasks; the paper reports point estimates without confidence intervals or significance tests; some baselines use different models, tools, or product interfaces, so they are not strict same-model harness ablations; experience base, global rules, and case-based adaptation lack full ablation across all tasks. The authors themselves enumerate these limitations.

A concurrent paper, "Rethinking the Evaluation of Harness Evolution for Agents," raises a broader evaluation issue: automatic harness evolution repeatedly uses task feedback and inference budget, so it must be compared against budget-matched parallel sampling, sequential correction, and other simple methods, while strictly separating search tasks from final evaluation tasks; otherwise gains may stem from more attempts or overfitting to specific benchmarks. This work is not a direct replication or refutation of MemoHarness, but it sets a higher standard for judging such research.

Conclusion: The Model's Brain Unchanged, But Its "Operating System" Upgrades

We used to view the model as the agent's brain and tools, memory, and workflows as auxiliaries. MemoHarness reminds us: as base model capabilities converge, what determines whether an agent works stably in complex environments may increasingly be not how much knowledge the model internally memorizes, but whether the external system can observe failures, distill experience, and change its runtime behavior accordingly.

Future agent evolution need not mean frequently training a larger model. It may appear as context strategies continually compressed, tool-calling rules continuously corrected, workflows re-orchestrated based on failure experience, memory learning to retain key states and discard stale information, and output validators patching new loopholes.

From this perspective, what enterprises should build long-term is not just model capability but a versionable, testable, rollbackable harness system that accumulates execution experience. The model's brain can stay the same, but the "operating system" around it can keep upgrading.

The part of an agent that truly evolves may not be the model. At least, the model is no longer the only part that can evolve.

References

[1] Yue Huang et al. "MemoHarness: Agent Harnesses That Learn from Experience." arXiv:2607.14159, 2026-07-14. (https://arxiv.org/abs/2607.14159)

[2] MemoHarness official code repository and method description. (https://github.com/HowieHwong/MemoHarness)

[3] Yike Wang et al. "Rethinking the Evaluation of Harness Evolution for Agents." arXiv:2607.12227, 2026-07-14. (https://arxiv.org/abs/2607.12227)