MemoHarness: The Next Evolution of Agents Happens Outside the Model

When people talk about "self‑evolving agents" they usually focus on changing the model itself—through self‑reflection, continued training, or reinforcement‑learning updates. MemoHarness proposes a different route: the model stays unchanged, while an external control system—called the Agent Harness—governs what context the model sees, which tools it may call, how steps are orchestrated, what memory is retained, and how results are validated.

The paper decomposes the Harness into six editable dimensions:

D1 Context Assembly : construct the input before each call, including instructions, constraints, retrieved materials, and examples.

D2 Tool Interaction : decide when to invoke external tools or retrieval and how many results to consider.

D3 Generation Control : adjust generation parameters such as max tokens, temperature, and sampling strategy.

D4 Task Orchestration : choose between a single deterministic call or a multi‑stage "plan‑execute‑revise" workflow.

D5 Memory Management : preserve key state across calls, summarize trajectories, and discard stale context.

D6 Output Processing : extract answers, verify format, and apply fallback strategies on failure.

During development, the system searches from a minimal initial Harness (no examples, no tools, deterministic single call) and evaluates candidate Harnesses on tasks with reference answers. Candidates are ranked primarily by correctness; token consumption is used only as a tie‑breaker, preventing convergence to cheap but inaccurate configurations.

Each execution records a rich case‑level experience: task features, the current Harness, the delta from the previous configuration, the full model‑tool trajectory, result score, token usage, and diagnosed failure modes. Periodically, the system extracts global patterns from many cases, identifying which strategies tend to succeed or fail and how the six dimensions interact, forming a two‑level experience base.

Evaluation on three benchmark suites demonstrates improvements:

Terminal‑Bench: 0.722 → 0.806

LiveCodeBench: 0.900 → 0.967

FinanceAgent: 0.600 → 0.767

The gains are most pronounced on long‑running, tool‑heavy tasks (FinanceAgent), suggesting Harness optimization is especially valuable for complex workflows. However, the authors note that the experimental scale is modest, statistical robustness is not fully validated, and cross‑task transfer is selective—not a universal Harness works for all tasks.

Cross‑dataset transfer experiments show mixed results: some downstream benchmarks (MMMLU, StrongReject, SWE‑Bench Pro) benefit, while others (HumanEvalFix, Reasoning‑Gym‑Easy, LawBench) see no change. Cross‑model tests with six additional models yield an average success‑rate increase of +0.098, indicating that certain control strategies may be model‑agnostic, though further rigorous testing is needed.

Cost analysis on the 18 held‑out Terminal‑Bench tasks reports $6.89 total (≈141.8 M input tokens, 1.332 M cached), cheaper than comparable Codex and Claude Code baselines but more expensive than OpenCode. The advantage hinges on high cache‑hit rates; without stable caching the cost benefit could disappear.

The authors acknowledge several limitations: the development‑phase search relies on labeled reference answers, the test‑phase does not update the experience base online, and the reported numbers lack confidence intervals or strict ablations. A companion paper on Harness evaluation highlights the need for careful baselines to avoid over‑fitting to budget‑driven search.

In conclusion, MemoHarness shifts the locus of agent evolution from model weights to the surrounding "operating system". By versioning, testing, and rolling back Harness configurations, enterprises can accumulate reusable execution experience that outlives any single model, making the external control layer a critical engineering asset for robust, controllable AI agents.

Agent evolution may no longer be about the model; the model is no longer the only part that can evolve.

References:

Yue Huang et al., "MemoHarness: Agent Harnesses That Learn from Experience," arXiv:2607.14159 (2026).

MemoHarness official repository, https://github.com/HowieHwong/MemoHarness.

Yike Wang et al., "Rethinking the Evaluation of Harness Evolution for Agents," arXiv:2607.12227 (2026).