Skip to content
VLM Memory Survey: 4D Taxonomy (When, Where, What, How) for Memory Mechanisms

VLM Memory Survey: 4D Taxonomy (When, Where, What, How) for Memory Mechanisms

Researchers from Duke, Google, USC, UPenn, UNC and others present a systematic survey of memory mechanisms in Vision-Language Models, introducing a 4D taxonomy (When, Where, What, How) covering 300+ papers, 70 benchmarks, 8 memory forms, and 8 application scenarios including long-video understanding and embodied navigation.

Machine Heart
Machine Heart
Machine Heart

Why VLMs Need Memory

Vision-Language Models (VLMs) have advanced rapidly in image captioning, visual question answering, video understanding, multimodal dialogue, and embodied intelligence. As tasks grow more complex and long-horizon, models must accumulate and recall information over longer time spans and more interaction rounds. Simply expanding the context window often fails to meet this need, prompting researchers to introduce memory mechanisms such as external memory banks, hierarchical memory, and short-term/long-term memory. However, existing designs are task-specific and lack a unified perspective for comparison. Prior surveys focus mainly on LLMs, leaving VLM memory mechanisms underexplored.

Core Framework: 4D Memory Taxonomy

The authors propose a 4D taxonomy that views memory as a full lifecycle: information is observed from multimodal tasks, written into memory, refined and organized over time, and retrieved during task processing. The four dimensions are:

When – Temporal Scope: How long memory persists. Transient memory serves single inference; episodic memory supports multi-turn dialogue/interaction (both last only during current reasoning). Persistent memory survives session ends for future interactions.

Where – Storage Location: Internal memory (model parameters, hidden states, KV cache) offers fast access and tight integration but is hard to modify or inspect. External memory (databases, knowledge graphs, experience banks) enables updates and auditing but depends on retrieval quality.

What – Memory Content: Raw information fragments, structured dialogue/trajectory records, abstract semantic knowledge, or compressed latent representations.

How – Memory Operations: Four operation types: Write (decide what new information to store and in what form), Consolidate (organize and refine existing memories), Retrieve (find task-relevant memories), and Update/Forget (maintain currency by updating and cleaning information).

Figure 2: 4D taxonomy framework for VLM memory: When, Where, What, How
Figure 2: 4D taxonomy framework for VLM memory: When, Where, What, How

Eight Typical Memory Forms

The four dimensions are conceptually separable but often bound in real systems. The authors identify eight typical memory forms, each a characteristic combination of the 4D choices. For example, long-context memory retains tokens or compressed representations via the context window (transient, internal), while lifelong agent memory accumulates experience across sessions (persistent, with all four operations considered). These combinations reflect common design tendencies, not fixed rules; e.g., transient intermediate states can be post-processed into episodic or persistent memory. The framework helps clarify existing preferences and highlight unexplored design spaces.

Figure 3: Eight typical memory forms and their positioning in the 4D framework
Figure 3: Eight typical memory forms and their positioning in the 4D framework

Eight Application Scenarios and Memory Preferences

The survey organizes nearly 70 VLM memory benchmarks and analyzes memory needs across eight scenarios:

Long Video Understanding & Generation: Context cannot hold massive frames; memory compresses visual information and selectively retains task-relevant visual cues.

Image Reasoning & Captioning: Models forget key details during long reasoning; memory preserves critical visual information.

Continual Learning: Core challenge is retaining old capabilities while learning new skills; memory protects original parameters and representation spaces.

Embodied Navigation & Action: Memory records action trajectories, task progress, and environmental spatial structure.

Multimodal RAG: Focuses on building external knowledge bases and document images, and on efficient, accurate retrieval.

First-Person Vision: Involves days or weeks of data; hierarchical memory organization is a key solution.

Personalized Assistants: User preferences span sessions; memory relies on updatable external stores.

Virtual Environment Interaction: Agents distill past interaction experience into reusable knowledge/skills for new tasks.

Figure 4: Memory requirement preferences across eight application scenarios
Figure 4: Memory requirement preferences across eight application scenarios

Challenges and Future Directions

The authors outline key challenges:

Longer context ≠ effective memory; models may still miss brief but critical events.

Information importance depends on future queries (lag), making compression/retention trade-offs hard; current methods often lose event order and visual details while keeping high-level semantics.

Retrieved memories can be irrelevant, outdated, or wrong; models lack verification ability and may confidently produce plausible but incorrect answers.

Benchmarks test single memory capabilities, but real-world errors span multiple stages; single-metric tests don't reflect deployed performance.

Promising research directions include:

Hierarchical memory structures inspired by human cognition.

Adaptive storage selection based on information type.

Models that learn to judge what is worth remembering.

Traceability: every memory linked to original visual evidence.

Evaluation beyond accuracy: user control (view, modify, delete memory) as a design requirement.

References

Paper: Memory in Vision-Language Models: Taxonomy, Mechanisms, and Applications

Authors: SHAO-JUN XIA, YIZHUO HE, YUNER ZHANG, YIFAN JIANG, XIAOYANG CHEN, LIANGXI LIU, CHENWEN LUO, JINBAO WANG

Project: https://github.com/Xzcv-hub/Awesome-Memory-in-VLM Preprint:

https://www.preprints.org/manuscript/202607.1539
Original Source

Signed-in readers can open the original source through BestHub's protected redirect.

Sign in to view source
Republication Notice

This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactadmin@besthub.devand we will review it promptly.

Vision-Language ModelsSurveyMultimodal RAGMemory MechanismsEmbodied NavigationLong Video Understanding4D TaxonomyPersonalized Assistants
Reader feedback

How this landed with the community

Sign in to like

Rate this article

Was this worth your time?

Sign in to rate
Discussion

0 Comments

Thoughtful readers leave field notes, pushback, and hard-won operational detail here.