How to Build Cross-Session Memory for RAG Chatbots: Short‑Term vs Long‑Term Strategies

1. Role of the Memory Module in Retrieval‑Augmented Generation (RAG)

A memory module acts as a second retrieval source in a RAG pipeline. The first source is a static knowledge base shared by all users; the second source is a dynamic, user‑ or session‑bound memory store. When a query arrives, the system simultaneously retrieves relevant static documents and relevant historical dialogue fragments, merges the two contexts, feeds the combined prompt to the LLM, and finally writes the new turn back to the memory store. This loop enables cross‑turn personalization such as recalling a policy number or dietary restriction.

2. Short‑Term vs. Long‑Term Memory

Short‑Term Memory (in‑session context)

Short‑term memory must answer “what was said earlier in this conversation?”. Three common strategies are combined in practice:

Buffer (full storage) : Accumulate every message verbatim and include the whole buffer in the prompt. Simple but quickly exceeds the LLM’s token window (e.g., 4 k‑8 k tokens).

Sliding window : Keep only the most recent N turns (e.g., N=5) and discard older turns. Prevents token overflow but loses older references.

Summary compression : Periodically summarise earlier turns into a concise paragraph and replace the raw messages with the summary. Saves tokens while preserving the gist, at the risk of losing exact numeric details.

Typical production pattern: retain the last 3‑5 turns in full, summarise older turns, and truncate the earliest content once a length threshold (e.g., 3 k tokens) is reached. This mirrors LangChain’s ConversationSummaryBufferMemory implementation.

Long‑Term Memory (cross‑session persistence)

Long‑term memory must answer “does the system still remember what the user said days ago?”. It requires solving three sub‑problems:

Location : Choose a persistent store. Common options are:

Redis – fast, in‑memory, suitable for hot recent data (non‑persistent by default).

MongoDB or PostgreSQL – durable, indexed by user ID, suitable for full history.

Vector databases (e.g., Pinecone, Milvus, Qdrant) – support semantic similarity search on embedded representations.

Real‑world systems often combine them: Redis for the most recent detailed turns, a relational DB for the complete log, and a vector store for semantic retrieval.

Storage format : Instead of dumping every utterance, apply one or more of the following:

Topic‑based partitioning – store different conversation topics in separate collections to avoid mixing unrelated contexts.

Fact extraction – parse dialogues to extract key facts (e.g., "user has diabetes", "prefers Japanese food") and store them as structured records.

Hierarchical summarisation – keep recent details, compress older turns into summaries, and further abstract very old data into high‑level overviews.

Retrieval : At the start of a new turn, encode the current query into an embedding (e.g., OpenAI text‑embedding‑ada‑002 or Sentence‑Transformers) and perform a similarity search against the long‑term vector store. The top‑k most relevant historical facts are injected into the prompt, exactly like static knowledge‑base retrieval but targeting user‑specific dialogue fragments.