Understanding Agent Memory: From Stateless LLMs to Persistent Multi‑Layer Architecture

Misguided Mental Model

Many people initially think that extending an agent's context window by simply stuffing more data solves memory problems, but context windows are limited to roughly 1‑2 million tokens and degrade when handling deep, long‑range information. Moreover, context is transient and disappears after a session, making continuous agents impractical without a persistent solution.

Four Types of Agent Memory

Agent memory is not a single concept; it consists of four layers serving different purposes:

Working Memory : The current context window containing user messages, dialogue history, and injected documents or tool results. It is fast but temporary, vanishing when the session ends.

Situational Memory : Records past conversations, completed tasks, decisions, and reasons. Stored externally and retrieved on demand, akin to an agent's diary.

Semantic Memory : Persistent factual knowledge such as user names, preferences, roles, or technology stacks. It is independent of any specific dialogue and resembles a user profile or knowledge base.

Procedural Memory : Encodes "how to do" information—available tools, workflows, and system prompts. Model weights themselves can be viewed as a form of procedural memory, containing billions of parameters that define reasoning and response generation.

These layers map to technical stack components: working memory to the context window; situational and semantic memory to external databases (vector stores, relational or key‑value stores); procedural memory to model weights and system prompts.

Operational Mechanisms

Write

At session end or key checkpoints, the agent (or a dedicated memory manager) decides which information to retain. Because storing everything would slow retrieval, only decisions, preferences, and non‑trivial context are kept. Typically an extra LLM call summarizes the raw dialogue into structured memory entries—facts, observations, events—then writes them to a vector database or key‑value store with timestamps and metadata.

Retrieve

When a new session starts or a new task arrives, the agent queries the memory store. Vector search is common: the current query is embedded, and the most semantically similar stored memories are fetched. Retrieved items are injected into the context window, giving the agent access to past experiences without rereading every historic conversation.

Forget

Forgetting is essential but often overlooked. Memory systems need decay mechanisms: low‑relevance items should fade, and contradictory entries (e.g., a preference switch from Python to Go) must be cleaned to avoid stale or inconsistent knowledge. Implementations use explicit time‑based expiration, recency/frequency signals, or more sophisticated spaced‑repetition‑style decay where frequently accessed memories stay active while dormant ones gradually disappear.

Current Progress

Agent memory is rapidly evolving, with several notable projects:

Mem0 : A widely adopted memory layer that sits between agents and databases, handling write, retrieval, and forgetting automatically. Its API offers simple save and search operations.

Letta (formerly MemGPT): Gives agents explicit control over memory via tool calls, allowing agents to decide when and what to write or delete. This increases complexity but also transparency.

Frameworks such as LangChain and LlamaIndex provide built‑in memory modules, and emerging model providers are embedding persistence directly into their APIs.

Direction is clear: memory is moving from ad‑hoc engineering solutions to a first‑class citizen in the agent stack with standard interfaces and dedicated infrastructure.

Relation to MCP and A2A

Model Context Protocol (MCP) addresses how agents access tools and data at the present moment, while Agent‑to‑Agent (A2A) handles inter‑agent collaboration. Memory complements them by preserving continuity over time, storing what happened in previous interactions.

Conclusion

Although LLM context windows grow and models improve, memory remains an engineered system comprising write, retrieval, and decay logic that does not automatically emerge from larger models. Designing and building this memory layer is the current engineering challenge worth attention.

by Nisarg Bhatt