Why Long-Term Memory Needs Vision: How MemEye Evaluates Multimodal Agent Recall

1. Why MemEye?

Most existing memory evaluations focus on text: remembering what a user said, retrieving past dialogue, or answering factual questions. In real situations, memory often comes from vision—for example, the position of a sofa in a home‑design plan, the later turn of a vehicle in a navigation video, or character changes in a comic storyline. MemEye addresses this gap by assessing whether models can retain and retrieve key visual evidence over long, multi‑turn, image‑rich conversations.

2. Two Axes: Visual Granularity and Reasoning Depth

MemEye evaluates agents along two dimensions. The first axis, visual evidence granularity , ranges from scene‑level understanding to pixel‑level discrimination, distinguishing tasks that require only overall context from those needing precise instance, location, or subtle visual differences. The second axis, memory reasoning depth , spans single‑point retrieval to relational inference and temporal evolution, separating questions that need a single recalled frame from those that compare multiple time points to judge state changes. This design moves evaluation beyond a single overall score, revealing whether a model is weak because it "doesn't see finely" or because it "fails to stitch information together".

3. Key Finding: Images Are Not Just Text Summaries

Experiments show that captions remain competitive for scene‑level and region‑level questions, but they fall behind significantly when the required evidence is at the instance or pixel level. Semantic retrieval often mistakes "relevant" for "valid", ranking outdated memories ahead of the correct answer. In other words, multimodal memory is not solved by converting an image into a single caption; the real challenge is locating the correct visual evidence when a later question appears and understanding its relationship to historical changes.

MemEye’s goal is to provide a finer‑grained, more diagnostic testbed for these capabilities.

https://github.com/MinghoKwok/MemEye
https://huggingface.co/datasets/MemEyeBench/MemEye
https://arxiv.org/abs/2605.15128