Why Agent Memory Needs AML: Inside AML’s Variable‑Control Engineering Design

Background and Issues with Traditional Agent‑Memory Benchmarks

Traditional agent‑memory evaluations have long been clouded by three sources of inflation: (1) downstream reasoning models that mask retrieval failures, (2) judge bias and over‑fitted prompting, and (3) self‑selected test sets that hide out‑lier failures.

AML’s Engineering Design

AML reconstructs the evaluation pipeline with four hard‑core designs that eliminate the above inflation.

Interface Responsibility Isolation (Add / Search Boundary) : Participants may only call two HTTP APIs—Add for ingesting long‑term dialogue/code and Search for returning top‑K evidence fragments.

Evaluation Contracts Version Hashing : All hyper‑parameters (e.g., top_k=100, timeout thresholds, fixed use of gpt‑4o‑mini for answer generation) are locked via a hash, preventing silent mid‑run changes.

Adaptive Concurrency & Resilience : The platform runs a 64‑worker asynchronous scheduler that automatically retries 5xx errors and rate‑limits from vendor APIs, ensuring stability under high‑pressure blind testing.

~5k‑Scale Private Blind Dataset : Built from 1.5 billion characters, the dataset contains nearly 5 000 blind questions covering seven core dimensions (recall, utilization, safety, etc.) with a public subset and a hidden private portion, providing high coverage without redundancy.

First‑Round Leaderboard Results

In the initial AML release, the text‑memory track separates commercial and open‑source tracks, applying identical test suites and evaluation criteria.

Commercial Track :

🥇 MemoraX – overall score 58.02, leading all seven text‑ability sub‑metrics.

🥈 MemOS – overall score 45.89, notable for low response latency.

🥉 NTES‑MEMORY‑SMART – overall score 44.21, demonstrating a solid technical foundation.

All top commercial systems are labeled “Fast” for total latency, write, and retrieval, and they exhibit leading token‑efficiency.

Open‑Source Track :

🥇 InvMem – overall score 45.06, combines knowledge‑graph and implicit indexing.

🥈 ReFind – overall score 44.97, narrowly trailing InvMem.

🥉 ActiveMemoryIndex – overall score 44.84, excels in dynamic memory re‑ordering.

Open‑Source Track Analysis

The close scores (a 0.22 gap between first and third) reveal that no single architecture dominates; instead, each approach shows asymmetric advantages for specific scenarios.

Fact Retrieval (baseline) : MemMachine, M‑flow, and Mem0 lead, confirming that robust evidence recall remains essential regardless of higher‑level reasoning.

Time Reasoning (bottleneck) : The best time‑reasoning score is 37.9, far below the top combined‑reasoning score of 53.1, indicating state‑transition and plan‑change inference as the industry’s most error‑prone area.

Memory Governance (update & forget) : ActiveMemoryIndex tops new‑value overwrite (54.50) and conflict resolution (24.79); Mem0 excels at long‑history summarization (65.00), showing the importance of dynamic correction and forgetting.

Personalization (graph exploration) : M‑flow wins with an Episode‑Facet‑Entity graph path score of 58.32, offering strong multi‑hop preference linking.

Rule Execution (infrastructure) : memfusion (overall rank 30) and InvMem capture the top two spots in the execution sub‑item (30.9 / 30.8), demonstrating hard‑core capability to turn history into workflow constraints.

Safety Boundary (extreme samples) : The lower‑ranked agent‑memory‑leaderboard scores a 90.36 no‑evidence‑response rate and a 41.67 minimal disclosure rate, leading the safety sub‑score (67.7) and providing a conservative reference for high‑compliance domains.

Overall, systems that win by engineering balance, rather than raw total score, reveal nuanced trade‑offs that are more informative for architecture selection.

Closed‑Source Track Analysis

MemoraX (v0.5) dominates with a total score of 58.0, a 12.1‑point lead over the runner‑up. It achieves Fast latency while securing first place across explicit recall (89.9), combined reasoning (63.4), time reasoning (60.0), memory governance (51.2) and context execution (30.0). MemOS follows with 45.9, leveraging strong fact retrieval (68.9) and time reasoning (56.5).

The middle tier is tightly clustered (41–44 points) with noticeable “specialization” patterns: NTES‑MEMORY‑SMART, AML‑FLASH, Cognee, TencentDB, and Mem0 each excel in particular sub‑metrics such as personalization or care‑related items. Commercial solutions generally mature in fact retrieval and personalization but lag behind in time/event reasoning (most scores 10–25) and workflow rule execution (many below 10), highlighting the next competitive frontier: converting memory into cross‑cycle dynamic state tracking and deterministic action rules.

Why AML’s Variable‑Control Matters

AML’s true contribution lies not in a single ranking but in turning previously hidden, easily‑cheatable evaluation variables into versioned, auditable contracts. For a field still defining common terminology and infrastructure, this engineered “variable control” provides a shared, transparent evidence base that enables fair technical discussion and systematic engineering progress.