How to Design an Effective Memory Module for LLM Agents?

1. Why Memory Matters for Agents

Most demo agents run a simple ReAct loop without memory, but in real business scenarios an agent that cannot recall a user’s preference from five minutes ago or repeats the same mistake is almost unusable. The design quality of the memory module largely determines the jump from "usable" to "good".

1.1 What Should an Agent Remember?

The first step is to decide the kinds of information the agent must retain. The author divides them into four categories, each requiring different storage and retrieval strategies:

Working Memory – short‑lived context, intermediate reasoning states, and tool results; implemented via the LLM's context window.

Episodic Memory – concrete past events with timestamps, such as a user’s seat preference or a failed API call.

Semantic Memory – abstract knowledge distilled from experiences, e.g., "the user prefers a minimal style" or "payment APIs need idempotent checks".

Procedural Memory – structured SOPs or workflows, like the step‑by‑step refund process.

These four types flow into each other: important fragments from working memory become episodic memories, repeated episodic memories are refined into semantic memories, and frequently executed procedures solidify into procedural memory.

1.2 Writing Memory

The biggest pitfall is "full‑recording" – dumping every dialogue turn into a database, which quickly inflates the store and drowns useful signals. A proper write pipeline follows perception → judgment → refinement → storage . Only new, relevant information triggers the pipeline; an LLM extracts a structured entry, checks for conflicts with existing memories, and then stores it.

1.3 Retrieving Memory

Retrieval decides "what to recall". Pure vector similarity is insufficient. Inspired by the Generative Agents paper, the author uses a three‑dimensional scoring model:

Recency – recent memories get higher scores, implemented with an exponential decay function.

Relevance – semantic similarity between the query and memory, measured by cosine similarity of embeddings.

Importance – intrinsic priority (e.g., VIP status) assigned by the LLM at write time or adjusted dynamically based on access frequency.

The final score is a weighted sum: score = α × recency + β × relevance + γ × importance Weights can be tuned per scenario (higher recency for customer‑service bots, higher relevance for knowledge‑base QA). Additional optimizations include metadata pre‑filtering (user ID, memory type, time range) and a two‑stage retrieval: coarse vector top‑50 followed by cross‑encoder re‑ranking to obtain the final top‑5.

1.4 Reflection and Integration

Beyond storage and retrieval, a reflection mechanism periodically lets the LLM review recent episodic memories, extract higher‑level insights, and store them as semantic memories. For example, after handling fifty return requests, the agent may infer that "product description mismatches are the main cause of returns" and use this insight to improve future interactions.

Other integration steps are memory merging/deduplication (clustering similar entries) and controlled forgetting. Forgetting can be achieved by decaying visibility scores for rarely accessed memories or by having the LLM flag outdated entries for archival or deletion.

1.5 Deployable Architecture

L1: Working Memory Layer – the LLM's context window, managed with a hybrid "summary + buffer" strategy: keep the last 3‑5 turns raw, summarize older turns, and maintain a structured task_state JSON for multi‑step reasoning.

L2: Recent Memory Layer – an in‑memory store such as Redis holding the full session history and high‑frequency memories from the past week, organized as a sorted set for fast range queries.

L3: Long‑Term Memory Layer – persistent storage combining a vector database (Milvus/Chroma) for semantic and episodic memories, a relational database (PostgreSQL) for structured user profiles, and optionally a knowledge graph (Neo4j) for complex entity relations. The three‑dimensional scoring occurs primarily at this layer.

Data flows bidirectionally: important information sinks from L1 to L2/L3, while relevant long‑term memories are loaded back into L1 at the start of a new task. Reflection runs on L3 to promote episodic memories into semantic ones.

1.6 Key Trade‑offs

Granularity – storing every sentence leads to rapid bloat; storing only high‑level summaries loses detail. The author recommends a two‑tier approach: summarized entries for routine retrieval and raw dialogue archives for occasional deep dives.

Privacy vs Personalization – user data must be deletable and isolated to comply with GDPR; providing users with view/edit/delete controls is both a legal and UX requirement.

Trustworthiness – LLM‑generated memory entries may be inaccurate. In high‑risk domains (medical, finance) the system should attach confidence scores, allow user verification, and give higher weight to high‑confidence entries during retrieval.