How to Handle Long Conversation History: Beyond Full Prompt or Recent Rounds

Answer in 30 seconds

Conversation memory is not about stuffing the entire chat into the prompt nor just building a vector store and stopping; information should be divided into three categories: static knowledge (policies, manuals), short‑term memory (recent conversation events), and long‑term memory (cross‑session facts that are valuable and allowed to persist).

Correcting a concept: chat logs are not memory

Chat logs are raw events (who said what and when). Memory is the curated information extracted from these events for later tasks. Conflating the two leads to two extremes.

Extreme 1: Full concatenation – token count grows, old topics and stale conditions interfere with current queries.

Extreme 2: Keeping only the recent few rounds – important earlier information may be lost, e.g., a budget confirmed early in the conversation.

Thus memory management solves four concrete questions: what to retain, where to store it, when to retrieve it, and when to update or delete it.

Three memory layers solve different problems

Short‑term window: ensure current task coherence

Stores recent raw messages, current task state, and nearby context, typically retrieved by session ID. It is raw and fast to update, but limited in capacity; older information naturally expires.

Window size should consider message count, token budget, and importance, prioritizing recent user inputs, pending task states, and confirmed items while discarding small talk or data that can be fetched from the knowledge base.

Short‑term data usually needs a TTL; the exact duration depends on product requirements, login status, data sensitivity, and compliance.

Summary layer: compress old history without creating new facts

When the raw conversation exceeds the window, earlier messages can be compressed into a summary that captures confirmed goals, key constraints, progress, unresolved issues, and important decisions.

Free‑text summaries risk missing negations, numbers, or exceptions; high‑risk fields should be stored structurally with a link to the original message.

Summaries should be versioned, timestamped, and linked to source messages; on conflict, the system should refer back to the original.

Long‑term memory: retrieve truly useful cross‑session information

Stores valuable cross‑session facts in persistent databases and optionally builds vector indexes for semantic search. Databases excel at precise filtering; vector indexes excel at semantic relevance. Both should be combined, but vector similarity must not replace permission checks.

Long‑term memory is not permanent; preferences change, tasks end, and facts expire. Each entry must have scope and lifecycle.

A memory’s seven-step lifecycle

Step 1: Decide if it’s worth writing

Not every message should become long‑term memory; trivial utterances like “thanks” lack value. The write strategy must check impact on future tasks, explicit user confirmation, duplication, sensitivity, and permission.

Model‑inferred preferences should be treated more cautiously than explicit user statements.

Step 2: Standardize and retain source metadata

A minimal memory record includes:

memory_id        // globally unique ID
owner_id         // user or organization
session_id       // originating session
topic/task_id    // related topic or task
type             // preference, fact, task state, summary, etc.
content          // usable content
source_ids       // original message or business record IDs
status           // active, superseded, expired, deleted
created_at       // creation time
expires_at       // expiration time (if any)
sensitivity      // sensitivity level
version          // current version

These fields answer who owns the memory, its source, current validity, and how to locate it for audit or deletion.

Step 3: Choose storage layer

Recent messages go to short‑term storage; auditable state goes to persistent DB; semantic‑searchable content gets a vector representation. The same memory_id must link across raw record, DB, vector index, and cache to ensure consistent updates and deletions.

If authoritative business data exists, do not duplicate it as long‑term memory; use the source directly.

Step 4: Scope‑filter before retrieval

When a request arrives, the system first determines user, organization, session, and task scope, then reads short‑term and candidate long‑term entries. Permissions are enforced before any semantic similarity ranking.

Step 5: Resolve conflicts during fusion

After retrieval, the system checks timestamps, status, topic, and source. New user‑confirmed values supersede older ones; conflicts between personal memory and authoritative knowledge favor the latter. The prompt should clearly separate “current question”, “knowledge base evidence”, “session state”, and “long‑term memory”.

Step 6: Update status after use

Usage can log last‑access time and hit result, but correctness must be driven by explicit new evidence (user correction, task completion, system state change). Old entries may be marked superseded and linked to a new version rather than overwritten.

Step 7: Expire and delete

Temporary task states expire automatically; user‑initiated deletions must remove data from short‑term cache, persistent store, vector index, and any backups. Deletion must be idempotent and verifiable via reverse retrieval tests.

Write‑time decisions precede retrieval heuristics

Designing a memory system that starts with vector stores and embeddings often overlooks the fundamental question of what content qualifies for long‑term storage.

Writing too much increases noise, privacy risk, and deletion cost.

Isolation matters more than retrieval accuracy

Cross‑user leakage invalidates any recall metric. Systems must enforce organization, user, session, and task boundaries at every layer, including cache keys and vector partitions.

Recall cannot rely solely on semantic similarity

Four signals should guide ranking: semantic relevance, scope consistency, temporal/status validity, and source trustworthiness. Hard filters precede soft ranking.

How to evaluate a memory system, not just a demo

Evaluation sets should include scenarios such as recent reference, returning to old topics, preference updates, similar‑task interference, session expiration, deletion, and cross‑user attacks. Metrics cover correct recall, precision, misuse of expired data, residual recall after deletion, latency, token cost, and write cost.

What a reproducible memory trace should record

A minimal trace logs request scope, selected short‑term messages, long‑term candidates (by memory_id), filtered-out candidates and reasons, injected memories, summary version used, and any update or deletion actions.

Logs should avoid storing full sensitive content; instead store IDs, types, status, and sanitized summaries.

Was the information worth writing?

Are user, task, and status correct?

Was scope filtering complete?

Did semantic retrieval pick the right candidate?

Did conflict merging let old values overwrite new?

Did expiration or deletion miss any replica?

Did the model respect source priority?

Interview follow‑up questions

Why not feed all history to the large model?

History grows unbounded and stale information interferes; a layered approach (short‑term window, summary, long‑term memory) is needed.

Why can’t a summary replace the original message?

Summaries may omit negations, numbers, or exceptions; high‑risk fields should retain structured links to the original.

Is a vector store enough for long‑term memory?

No; vector similarity must be combined with user/task filtering, source/status checks, conflict resolution, expiration, and isolation.

How to handle changing user preferences?

New confirmations create a new version; the old entry is marked superseded and excluded from retrieval.

What should be deleted when a user clicks “clear”?

Product definition dictates whether to clear only the current session, all short‑term history, or also long‑term memories; implementation must sync cache, session store, DB, and vector index, then verify via reverse retrieval.

How to prevent memory cross‑talk between users?

Authenticate identity server‑side, enforce scope filters before retrieval, include scope in cache keys, and validate with negative tests.

How to prove memory effectiveness?

Beyond a single demo, use tests covering recent reference, topic return, preference update, similar tasks, expiration, deletion, and cross‑user leakage, measuring correct recall, erroneous injection, stale usage, and residual data.

Final answer recap

I would design conversation memory as a lifecycle‑managed context system rather than a raw chat array. Static knowledge comes from a knowledge base; short‑term window holds recent messages and task state; summaries compress older history; only cross‑session valuable facts enter long‑term memory.

Each memory record includes user, session, task, source, status, timestamps, sensitivity, and a unified ID. Retrieval first applies permission and scope filters, then selects by recency or semantic relevance. Fusion resolves conflicts, updates versions, handles expiration, and supports deletions that propagate across all storage layers and are verified by reverse retrieval.