Introducing the First Open Agent Memory Challenge – A Unified Benchmark for Long-Term Memory Systems

Unified Evaluation Design

Agent Memory Leaderboard (AML) integrates public benchmarks, private test sets, a standardized access protocol, a uniform answering workflow, and a governance framework into a continuously running evaluation ecosystem. Fairness is ensured by covering both text and code memory scenarios with a single data pipeline, a fixed answer model, and a consistent scoring process.

Text Memory Dataset

AML aggregates more than ten mainstream text‑memory datasets—including PersonaMem, LoCoMo‑Refined, CLBench, BEAM, LongMemEval, ScriptMem and others—covering over 1,500 dialogues and tasks, roughly 1.5 billion characters of long‑range history, and about 5,000 evaluation questions. This breadth reduces the risk of over‑optimising for a single scenario and yields results that better reflect performance in complex real‑world settings.

Code Memory Dataset

AML introduces a newly constructed code‑memory dataset (not yet public) derived from real GitHub repositories. The dataset spans 12 repositories and 150 base tasks; each base task is paired with 1,290 finely annotated historical pull‑request tasks that occurred earlier in time. Three relevance levels (strong, weak, unrelated) are encoded, and strict temporal ordering prevents future‑information leakage. Three memory environments with varying history size and noise density simulate diverse software‑development contexts.

Ability Profiling

Performance is decomposed into a multi‑dimensional ability profile that follows the memory‑usage chain: precise recall, correct understanding, and stable usage within appropriate boundaries. Text memory is mapped to seven dimensions: explicit factual recall, relational & multi‑hop composition, temporal & event sequencing, memory governance, personalization & care, rule & workflow execution, epistemology, and security & privacy. Code memory is evaluated through two task types: Debug Memory (retrieving past fixes for similar issues) and Development Memory (extracting architectural designs and engineering patterns from historical records).

Evaluation Process

Participating systems implement only two core interfaces: Add – writes memories into the system. Search – returns evidence relevant to a query.

The platform supplies a fixed Answer model to generate final responses, a unified Eval pipeline for scoring, and aggregation rules. Scoring uses a Multi‑Agent Judging System that routes questions to specialized reviewers based on ability tags: factual recall checks key facts, multi‑hop checks evidence integration, temporal questions check state transitions, security questions verify privacy boundaries, and code tasks are judged by test‑case pass/fail.

Low‑Barrier Access Protocol

The protocol reduces entry cost to the stable Add and Search endpoints. Commercial products can expose a stable API, while open‑source methods submit code and configuration for reproducible evaluation. The platform handles answer generation, scoring, result aggregation, and orchestration, enabling continuous inclusion of new systems and version‑tracked results.

Participation Workflow

Apply : submit a registration form to obtain an API key.

Integrate : expose public Add and Search endpoints following the AML protocol.

Submit : pass a compatibility smoke test, then submit the full evaluation task.

Verify : results are audited and, if approved, appear on the public leaderboard.

Resources

Leaderboard repository (protocol, submission guide, examples): https://github.com/AML-memory/agent-memory-leaderboard