MemoHarness: How Agent Evolution Shifts to the External System

Core Idea – The paper introduces Agent Harness , a six‑dimensional control layer that sits outside the LLM and governs what context the model sees, which tools it may call, how many reasoning steps are performed, how state is remembered, and how outputs are verified. By treating the entire harness as a learnable object, MemoHarness moves the target of “evolution” from model parameters to the external system.

Six Editable Dimensions

D1 Context Assembly : construct prompts, constraints, and retrieved examples; optionally compress context.

D2 Tool Interaction : decide when and how many external tools or retrieval results to invoke.

D3 Generation Control : adjust max tokens, temperature, sampling strategy, and budget.

D4 Task Orchestration : choose between single‑call or multi‑stage “plan‑execute‑revise” workflows.

D5 Memory Management : retain key state across calls, summarize trajectories, discard stale context.

D6 Output Processing : extract answers, validate format, and apply fallback strategies on failure.

The authors argue that many agent failures stem from mismatches in these external mechanisms rather than from the model itself.

Learning Process

MemoHarness separates development and testing phases. During development, a simple baseline harness (no examples, no tools, deterministic single call, no memory) is iteratively refined on tasks with reference answers. Each configuration is scored on correctness first, then token cost as a tie‑breaker, ensuring the search does not collapse to cheap but inaccurate solutions. Every run records task features, harness settings, model‑tool traces, scores, token usage, and diagnosed failure modes, forming a case‑level experience.

In the testing phase, the accumulated experience library is frozen. For a new, unlabeled task, the system retrieves similar successful and failed cases, applies global harness patterns, and adapts them to the specific task in a single adaptation step—no online feedback or further search.

Experimental Results

The authors evaluate three task families:

Terminal‑Bench (long‑horizon terminal operations) : success rate improves from 0.722 to 0.806.

LiveCodeBench (single‑shot code generation) : success rate improves from 0.900 to 0.967.

FinanceAgent (multi‑step tool‑heavy financial analysis) : success rate improves from 0.600 to 0.767.

Improvements are larger on tasks that require extensive tool use and memory, suggesting that harness optimization is most valuable for long‑running, tool‑intensive scenarios.

Cross‑task transfer is selective: the learned harness boosts performance on some benchmarks (MMMLU, StrongReject, SWE‑Bench Pro) but leaves others unchanged (HumanEvalFix, Reasoning‑Gym‑Easy, LawBench). Cross‑model transfer (applying a harness found with GPT‑5.3‑Codex to six other models) yields an average success‑rate increase of 0.098, with gains ranging from 0.038 to 0.233, indicating partial model‑agnostic benefits.

Cost analysis shows that, with a high cache‑hit rate, the Terminal‑Bench experiments cost $6.89 USD, cheaper than comparable baselines but dependent on reusable experience context.

Limitations and Future Work

The study acknowledges several constraints: limited number of evaluation tasks, lack of confidence intervals, heterogeneous baselines, and incomplete ablations of the experience library and global rules. The authors also note that the current system does not continuously ingest new experience during deployment; each test task receives a one‑time adaptation without feeding results back into the learning loop. Extending to online, lifelong evolution is left as future work.

Related work (Rethinking the Evaluation of Harness Evolution for Agents) stresses the need for rigorous baselines that match budget and sampling strategies, warning that observed gains might stem from extra attempts rather than genuine harness improvements.

Implications

MemoHarness demonstrates that substantial performance gains can be achieved by versioned, testable, and roll‑back‑able changes to the agent’s “operating system” while keeping the underlying model static. For enterprises, this suggests that long‑term competitive advantage may lie more in curated harness configurations, experience libraries, and diagnostic tooling than in ever‑larger LLMs.

Agent evolution may no longer be limited to the model; the surrounding system can evolve independently.