MemoHarness: The Next Evolution of Agents Happens Outside the Model

Overview

MemoHarness introduces the concept of an Agent Harness that keeps the underlying language model unchanged while continuously improving the surrounding control system that decides context, tool usage, workflow, memory, and output validation.

Six Editable Dimensions

D1 Context Assembly : constructs the input before each model call, allowing structured prompts, examples, or context compression.

D2 Tool Interaction : determines when to invoke external tools or retrieval modules and how much evidence to incorporate.

D3 Generation Control : adjusts decoding parameters such as maximum tokens, temperature, and sampling strategies.

D4 Task Orchestration : chooses between a single deterministic call and a multi‑stage “plan‑execute‑revise” workflow.

D5 Memory Management : stores cross‑call state, summarizes trajectories, and discards expired context.

D6 Output Processing : extracts answers, validates format, and applies fallback strategies on failure.

Training‑time Optimization

The paper treats “training‑time optimization” as a search over harness configurations on development tasks that have reference answers. The base model remains frozen; every modification occurs in the external control layer and is recorded for later analysis.

Experience Library

Each execution logs task features, the current harness, the delta from the previous configuration, the full model‑tool trace, result score, token consumption, and a diagnosis of the primary failure mode. These logs constitute case‑level experience, which is periodically distilled into global rules that relate the six dimensions.

Evaluation

MemoHarness was evaluated on three benchmark families:

Terminal‑Bench (long‑running terminal tasks): 0.722 → 0.806

LiveCodeBench (single‑call code generation): 0.900 → 0.967

FinanceAgent (multi‑step tool‑heavy analysis): 0.600 → 0.767

Improvements were larger on the tool‑intensive FinanceAgent, suggesting that harness optimization benefits complex pipelines more.

Cross‑task transfer showed selective gains: the learned harness helped on MMMLU, StrongReject, and SWE‑Bench Pro, but not on HumanEvalFix, Reasoning‑Gym‑Easy, or LawBench. Cross‑model transfer to six additional LLMs raised average success by +0.098 (range +0.038 to +0.233).

Cost Analysis

On the 18 held‑out Terminal‑Bench tasks, MemoHarness consumed 14.18 M input tokens (13.32 M cached) for a reported cost of $6.89, cheaper than Codex and Claude Code but higher than OpenCode. The advantage relies on high cache‑reuse; without it, cost dynamics change.

Limitations and Future Work

The authors acknowledge several constraints: limited evaluation scale, lack of confidence intervals, non‑uniform baselines, and incomplete ablations of the experience library. The current system does not continuously ingest new experience after deployment; online self‑evolution is left as future work.

A companion paper (Rethinking the Evaluation of Harness Evolution for Agents) stresses the need for stricter baselines that match budget and sampling strategies, warning that observed gains could stem from more attempts or benchmark over‑fitting.

Implications

MemoHarness shifts the locus of “agent evolution” from model weights to an external, versioned control stack that can be audited, rolled back, and cost‑optimized. For enterprises, this means that competitive advantage may derive more from curated harness configurations and experience libraries than from ever larger LLMs.

Agent evolution may happen outside the model; the model is no longer the sole component that can improve.