Designing Short‑Term and Long‑Term Memory for AI Agents: Key Strategies and Trade‑offs

Two‑Layer Memory Architecture

In practice, an agent’s memory is split into a short‑term layer for the current session and a long‑term layer for cross‑session context. The short‑term layer keeps the dialogue within the prompt length by truncating or summarising, while the long‑term layer retrieves relevant historical information via vector similarity and injects it into the context.

Short‑Term Memory Designs

1. Fixed‑Window Truncation

When information value decays quickly, keep only the most recent N turns or N tokens, discarding older content. This approach is simple, low‑cost, and keeps length stable, suitable for chatbots and basic Q&A. Its drawback is a blunt forgetting mechanism that can drop early crucial instructions.

2. Rolling Summarisation

Instead of discarding, summarise earlier dialogue into a concise summary and replace the original records when the context window fills up. Advantages: preserves high‑value items such as task goals, style constraints, and confirmed conclusions while reducing irrelevant details. Cost: an extra model call and the quality of the summary directly affects downstream performance.

Long‑Term Memory Construction

The long‑term layer solves cross‑session recall by storing each turn as an embedding in a vector database. Retrieval works in three steps: Store – embed and write the turn with its raw text; Retrieve – perform similarity search with the new query; Combine – feed the most relevant historical snippets together with the current question to the model.

Benefits: the agent is no longer limited by the prompt window and can access relevant history from a much larger span, enabling personalised assistants, enterprise knowledge bases, and long‑term learning companions. Drawbacks: higher system complexity, requiring embedding models, a vector store, and retrieval logic.

Typical content worth persisting includes stable user preferences, core task objectives, verified facts, and conclusions that will be reused in future interactions.

Memory Governance

Long‑term memory is a dynamic data asset; it needs periodic cleaning, merging duplicate entries, and fact‑checking. Providing interfaces for users to view, edit, or delete entries is essential for maintaining a stable memory system.