EvoMemBench: Why AI Agent Memory Must Evolve, Not Just Remember

The article opens with a motivating comparison: one agent scores 85% on its first task and stays at 85% after 100 tasks; another starts at 70% but reaches 80% by task 20 and 91% by task 100 by absorbing feedback, updating knowledge, and reusing effective experience. This illustrates the core thesis: doing a task right once is capability; improving over time is evolution, and Agent Memory is the mechanism that turns past interactions, states, and experiences into reusable information for future tasks.

Memory, Long Context, and RAG Are Not the Same

The author distinguishes three mechanisms in one sentence: long context answers "what just happened," RAG answers "what do external sources say," and Memory answers "what will be needed later." A concrete example shows the division of labor when a user asks to cancel a recent overseas order: long context identifies which order and what tool calls were made; RAG retrieves current cross-border refund policies and compliance rules; Memory supplies the customer's long-standing handling preferences, current compliance-review status, and whether the experience from this refund should be reused.

Four Questions for Building Memory

Real-world memory construction reduces to four questions: what to record (user preferences, business rules, tool results, task states), what form to store it in (summaries, tables, vectors, graphs, processes, skills), who consumes it (current task, a user, a business type, the whole agent team), and how to govern it (provenance, versioning, permissions, expiration, deletion, security). The hard problem is not storage but deciding what is worth recording, how to update it, when to retrieve it, and when to invalidate it.

From "Remembering" to "Evolving": The Benchmark Evolution

Early memory benchmarks focused on fact recall. Long-running agents demand more: correcting outdated preferences (e.g., a user switches from window to aisle seat after a leg injury), preserving precise execution state mid-task (which flight, which passenger, confirmed parameters, next step), and transferring reusable experience across tasks (API call sequences, source-selection strategies, checkpoint steps). The evaluation goal has shifted from retrieval to correction, from acting with memory to reflecting on experience, and toward active learning plus long-term cost, latency, deletion, privacy, and pollution defense.

EvoMemBench: A Unified Four-Quadrant Framework

Existing benchmarks test isolated capabilities (knowledge vs. execution, within-task vs. cross-task) with inconsistent models, context budgets, and update mechanisms, making cross-comparison hard. Zhejiang Chuanglin Technology and HKUST (Guangzhou) jointly developed EvoMemBench to unify these dimensions. Two axes create four quadrants:

INEP-KNOW (In-task Knowledge Evolution) : tests Retention (can history be retrieved?) and Revision (can conflicting new/old information be resolved to the current valid state).

INEP-EXEC (In-task Execution Evolution) : tracks execution state — e.g., "select the second option," "keep the same passenger," "change the order to Tuesday" — where losing a key entity or step state is fatal.

CROSSEP-KNOW (Cross-task Knowledge Evolution) : accumulates and recomposes reusable knowledge (e.g., corporate travel rules appearing in different tasks, later combined; when a unified rule updates, old conclusions must be replaced, not coexist as conflicts).

CROSSEP-EXEC (Cross-task Execution Evolution) : reuses execution experience, split into Tool Using (API patterns, parameters, preconditions), Web Search (source selection, cross-verification), and Embodied AI (object positions, action preconditions, environment conditions — past successes may not transfer).

The benchmark comprises 4 evolution capabilities, 6 benchmarks, ~5.8K samples : INEP-KNOW 2,800, INEP-EXEC 800, CROSSEP-KNOW 884, CROSSEP-TOOL 800, CROSSEP-WEB 270, CROSSEP-EMB 200.

Five Key Conclusions from Evaluation

1. When History Fits in Context, Raw Context Beats External Memory

In INEP-KNOW, when full history fits in context, Gemini-3-Flash reading raw context scores 98.2 (Event QA), 58.0 (FC-MH), 96.0 (FC-SH) ; the best external memory scores 91.2, 7.0, 54.0 respectively. External memory adds extraction, storage, recall, and re-injection steps — each can lose information or add noise. Memory only adds value when it fills a genuine model/context gap.

2. The Harder the Task, the More Memory Helps

Across 13 memory methods: on easy tasks 0/13 beat baseline (avg -6.09); on medium tasks 6/13 beat baseline (avg +0.32); on hard tasks 13/13 beat baseline (avg +7.12). Over-recall on simple tasks (e.g., pulling overseas approval, large-amount approval, executive approval for a question needing only one rule) can make the agent hesitate on an already correct answer. The goal is not maximal recall but supplying key information exactly where the model has a capability gap.

3. Memory's Value Shows Most Clearly in Long-Horizon Execution

Execution state differs from QA: shorter context windows truncate earlier tool results and intermediate states, so memory that preserves "what step we're on, key parameters, next action" becomes more valuable. At context sizes 16K, 32K, 64K, 128K , the number of memory methods beating the no-memory baseline were 11/15, 11/15, 12/15, 9/15 respectively. Whether to use memory must be judged per task, history length, and context budget.

4. Retrieving Old Information ≠ Updating Old Conclusions

The most alarming finding: Retention averages 51.6%, Revision only 18.1% . Revision is not simple overwrite; it requires a chain: retrieve relevant history, determine if it's the same entity, detect conflict, decide priority, then execute update, deprecation, or invalidation. A failure at any step yields a system that finds the old fact perfectly but still outputs the wrong current state. Long-term memory systems must treat Update, Invalidate, Rollback as first-class capabilities, not afterthoughts.

5. No Single Memory Type Fits All Tasks

By Average Rank (lower is better): Retrieval-based Memory excels in Tool Using (5.43) ; General Long-term Memory leads in Web Search (5.38) ; Procedural Memory wins in Embodied tasks (5.60) . Experience transfer mirrors this: Tool Using shows more positive transfer (decision steps like state-check, call-selection, result-verification reuse); Embodied tasks mix positive and negative transfer because environment, object positions, and preconditions change. What should transfer is not "the exact steps taken last time" but "how this class of task should be decided" plus the applicability conditions (environment, preconditions).

From Benchmark to Engineering: Five Construction Principles

Establish a strong baseline first. Use long context and simple retrieval to verify how far the task gets without complex memory. Only add complexity when memory delivers measurable gain.

Don't invoke memory by default. Memory should intervene only when model capability or context budget shows a gap, not as a fixed "retrieve history" step in every task.

Treat "update" as a core capability, not just "store" and "search." Update, Invalidate, and Rollback determine whether the system maintains a correct current state.

Distinguish "experience abstraction" from "precise state." Processes, preferences, decision patterns can be abstracted; order IDs, file paths, concrete entity parameters must be stored exactly. The two cannot be handled with one blunt approach.

Record applicability conditions for every reusable experience. Especially for execution experience, save environment config, preconditions, and scope of validity; otherwise a success in one environment becomes negative transfer in another.

Finally, the author proposes a net-value formula :

Memory Net Value = Task Gain - Token Cost - Retrieval Latency - Maintenance Cost - Negative Transfer Loss - Security Risk

. This frames the pragmatic judgment: long-running agents need not an ever-growing history warehouse but a mechanism that continuously decides what to keep, what has expired, and under what conditions reuse is valid.

The closing refrain: "Getting it right once is capability; getting better after doing it is evolution. Memory exists not to give the agent more history, but to give it better judgment, more accurate state, and more appropriate action the next time it faces a problem."