Why Can an Agent Remember You? Exploring Memory Types and Retrieval Routing

1. What an Agent Needs to Remember: Three Dimensions

Before designing a memory system, the article splits "memory" into three orthogonal axes: time axis (short‑term vs. long‑term), function axis (episodic, semantic, procedural), and source axis (static knowledge vs. dynamic experience). Each axis implies different storage forms, retrieval methods, and update policies.

2. Why Naïve Approaches Fail

The author reviews three common shortcuts:

Expanding the context window : works briefly but token cost grows linearly and LLMs lose attention to middle tokens.

Storing everything in a vector store : captures semantic similarity but discards temporal order and versioning, causing outdated facts to surface.

Appending logs without organization : full‑text search suffers from signal‑to‑noise decay and inability to determine the currently valid version of a fact.

All three share the flaw of treating memory as a single undifferentiated bucket without lifecycle management.

3. Four Core Modules of a Memory System

Module 1 – Extraction

Decides what content is worth remembering and how to pull it from the dialogue stream. Three strategies are described:

Raw sequence concatenation : cheapest, zero information loss, but low signal‑to‑noise and high token cost.

Unstructured semantic extraction : LLM generates fact statements; higher quality but may split context incorrectly. A "conservative extraction, late filtering" pattern mitigates this.

Structured extraction : LLM outputs schema‑conforming triples (e.g., (entity1, relation, entity2)) or timestamped events. Strongly typed, suitable for graph databases, but higher cost and requires validation.

An engineering lesson: higher compression reduces retrieval speed but sacrifices original evidence needed for precise citations.

Module 2 – Representation & Storage

The memory’s physical form can be:

Text sequence : human‑readable, easy for LLMs, but weak for relational queries.

Knowledge graph : timestamped triples enable conflict detection and temporal reasoning, at the expense of extraction complexity.

Hierarchical time tree : organizes memories by year/month/week/day, allowing O(log N) navigation for time‑sensitive queries.

Production systems often use a heterogeneous mix of these representations, with indexes treated as rebuildable accelerators rather than the source of truth.

Module 3 – Retrieval & Routing

Retrieval is the most complex part. Five mechanisms are listed, each suited to different query types:

Semantic vector search : finds semantically similar memories but lacks temporal awareness.

BM25 full‑text search : excels on exact keyword matches, especially for ticket IDs or precise terms.

Time‑hierarchy navigation : drills down from coarse to fine time buckets, ideal for queries like "last week’s architecture review".

Knowledge‑graph traversal : aggregates evidence spread across multiple sessions by following entity relationships.

LLM‑driven routing : the model decides which retrieval path(s) to take, offering maximum flexibility at higher inference cost.

Production solutions typically blend vector and BM25 results using reciprocal rank fusion (RRF) or learned weights.

Module 4 – Maintenance

Maintenance prevents the memory store from degrading into a noisy dump. Three sub‑problems are addressed:

Conflict resolution & versioning : use timestamped multi‑versioning ( valid_from / valid_to) or LLM‑driven UPDATE/DELETE calls to mark superseded facts.

Capacity management : apply hard limits (FIFO) or salience‑based eviction, where a salience score (see code below) ranks importance.

Semantic consolidation : periodically merge duplicate entries and delete stale content, but only when similarity exceeds a high threshold to avoid losing detail.

salience = density_score + kind_boost + pin_boost

density_score = min(content_length / 500, 1.0) * 0.45
kind_boost    = 0.20  # constraints / preferences / procedures / definitions
pin_boost     = 0.20  # user‑pinned importance

4. Cross‑Cutting Concerns

Salience

Not all memories are equal; a salience score attached at write‑time determines eviction priority, ranking weight in retrieval, and exemption from time‑decay for high‑signal types (constraints, preferences, procedures, definitions).

Prompt Stability vs. Memory Updates

LLM providers cache prompts when the prefix is unchanged, saving 70‑80% of input token cost. Injecting new memory into the prompt invalidates this cache. Two injection strategies are compared:

Frozen snapshot : core preferences (≤1500 tokens) are rendered once at session start and never changed, preserving cache hits.

On‑demand tool retrieval : historical details are fetched via tool calls, keeping the prompt stable while still accessing deep memory.

A table in the original article compared these approaches; the key take‑aways are reproduced as bullet points above.

5. Reference Architecture

The four modules and two cross‑cutting concerns combine into a production‑ready Agent Memory pipeline. Core design decisions include:

Classify at write‑time rather than guessing importance at query time.

Treat indexes as accelerators, not the source of truth.

Freeze prompt‑resident memory; use tool‑based retrieval for historical facts.

Make maintenance (consolidation, eviction, flush) a first‑class system component.

6. Common Failure Modes

Memory pollution : outdated but semantically similar entries are returned together, leading to mixed answers. Fix: filter by temporal validity during re‑ranking.

Memory omission : relevant evidence is scattered across sessions; a single vector query returns only the newest slice. Fix: hierarchical organization (time tree or graph) to aggregate evidence.

Maintenance explosion : global consolidation scales quadratically with memory size, exhausting token budgets. Fix: switch to localized consolidation of newly written items.

7. Workload‑Driven Selection Matrix

Different workloads favor different representations, retrieval methods, and maintenance strategies. Highlights:

Cross‑session aggregation (personal assistants, CRM): knowledge‑graph + timestamped versioning, graph + semantic mix, explicit conflict resolution.

Long‑form coherent dialogue (coding assistants): time‑hierarchy + text sequence, time navigation + BM25, layered retention with pre‑flush compression.

State‑dependent tasks (database workflows): full operation trace + light summary, time‑order retrieval, prioritize trajectory preservation.

High‑frequency short sessions (cost‑sensitive products): minimal resident text facts, on‑demand tool retrieval, frozen snapshot for prompt cache.

Continuous growth scenarios: layered hot‑cold storage, intent‑based routing, local consolidation with salience‑based eviction.

The final principle stresses identifying the bottleneck of the target workload before choosing an architecture, rather than applying a one‑size‑fits‑all solution.

Conclusion

Agents often mistake "memory" for mere storage. Effective systems must actively classify, index, retrieve, and prune information, mirroring how human memory works—by constantly organizing, correcting, and forgetting. The smarter an agent is at forgetting, the more reliable its answers become.