WorldMemArena: Benchmarking Multimodal Agent Memory in Action-World Loops

Researchers from UC Santa Barbara, Stanford, ETH Zurich, Johns Hopkins, Carnegie Mellon, and J.P. Morgan Chase introduce WorldMemArena , a benchmark for evaluating multimodal agent memory through continuous action–world interaction. The paper, accepted at NeurIPS 2026, is available on arXiv (https://arxiv.org/abs/2605.29341), with code on GitHub (https://github.com/UCSB-AI/WorldMemArena) and data on Hugging Face (https://huggingface.co/datasets/LCZZZZ/WorldMemArena).

Problem: Recall ≠ Useful Memory

Existing memory benchmarks often test static recall via final QA accuracy. A wrong answer can stem from failure to write key facts, failure to update outdated information, failure to retrieve stored evidence, or failure to use retrieved evidence. Compressing images to text also loses spatial, temporal, and procedural details.

Action–World Interaction Loop

WorldMemArena embeds memory evaluation in an Observation–Action–Environment Feedback loop. Agents face changing task states, web pages, files, screenshots, and tool results. The benchmark decomposes long-term memory into four diagnosable stages:

Write – select potentially useful evidence from current observations.

Maintain – update, merge, or discard outdated content when the world state changes.

Retrieve – fetch relevant evidence before the next decision.

Use – integrate evidence into the answer or action.

Tasks are annotated with key memory points, state updates, distractors, and evidence chains, enabling stage-level diagnosis.

Figure 1: Traditional benchmarks focus on static recall; WorldMemArena adds multimodal cues, memory lifecycle, and world changes after actions.

Figure 2: Memory flows through observation, update, retrieval, and action; actions change the world for the next round.

Dataset: Lifelong Evolution & Agentic Execution

WorldMemArena contains 400+ multi-session tasks, 24,258 QA pairs, and 15,595 images/screenshots . Two task categories:

Lifelong Evolution – tracks cross-session changes like user preferences and project progress.

Agentic Execution – sourced from real or reconstructed agent trajectories, preserving observations, operations, tool feedback, and visual states.

Agentic Execution tests action-oriented memory: agents must remember what they tried, which step failed, and how the environment changed. Checkpoints at intermediate and final stages cover fact recall, dynamic updates, conflict resolution, visual search, and cross-modal reasoning.

Figure 3: Extracting and updating key memory points from multi-session windows, building staged QA; tasks cover GUI and embodied interaction.

Key Findings: Storing More ≠ Using Better

Under a unified setup, the paper compares long-context models, RAG with external memory, and agent-managed memory harnesses. A core result: high memory recall does not guarantee high QA accuracy .

M2A : recall 86.83%, QA accuracy 50.14%.

MemGPT : recall 85.20%, QA accuracy 57.81%.

These numbers (from Table 2, sample-averaged percentages) show that "how much was written" ≠ "whether correct evidence is used at answer time".

Table 1: Memory quality and final QA performance are not always aligned. Values from paper Table 2 Recall and QA-C columns.

Additional findings:

Systems tend to append new memories rather than revise outdated ones.

Visual information enters the memory store but often fails to support cross-temporal, cross-modal reasoning.

Forgetting obsolete information is as critical as storing new information in a changing world.

Harder Tasks Expose Deeper Gaps

Cross-scenario analysis shows most systems perform better on Lifelong Evolution than on Agentic Execution. Visual search, visual state update, and cross-modal reasoning reveal a further weakness: agents may have "seen" screenshots but do not convert visual changes into usable evidence for subsequent decisions.

Figure 4: Performance differences across long-term state evolution, agent execution trajectories, and visual QA.

Implications

WorldMemArena shifts focus from "can the agent repeat the past?" to "can it maintain a continuously changing world and let past experience guide the next action?" Current results highlight multiple gaps between storage capacity, retrieval quantity, and final answer correctness. Reliable agent memory requires revision, evidence discrimination, and the ability to turn failures and feedback into judgment for the next step.