Designing AI Agent Memory: From Simple Context Concatenation to a Cognitive System (Part 1)

Why an Agent Needs Memory

Most agent systems treat memory as simple concatenation of past messages. This approach creates several concrete problems in realistic usage:

Linear context‑window growth – longer dialogs consume more tokens, making inference cost uncontrolled.

Cross‑session forgetting – users must repeat profession, preferences, background each time they start a new session.

Knowledge cannot evolve – old and new information coexist and may contradict (e.g., "user uses Python" vs. "user switched to Go").

Weak retrieval – keyword‑based matching limits semantic recall, making it hard to link related intents such as "prefer concise replies" and "please be brief".

Tool‑call risk – hard truncation can split AIMessage(tool_calls) from its corresponding ToolMessage, causing LLM‑API errors.

The root cause is storing raw data without a cognitive structure; the system remembers "what was said" but does not understand "who you are" or "what you want".

Core Capabilities for an Evolvable Cognitive Memory

Layered memory – separate management of information at different time scales (current dialogue, today’s session, month‑long preferences).

Semantic retrieval – retrieve memories based on intent rather than literal keyword matches.

Memory evolution – automatic deduplication, conflict resolution and dynamic updates when facts change.

User modelling – aggregate scattered facts and preferences into a coherent user profile.

L1–L3 Three‑Level Memory Framework

The design adapts the classic Atkinson‑Shiffrin model (1968), which splits human memory into sensory, short‑term (working) and long‑term components. Recent AI‑agent research (e.g., "Memory in the Era of AI Agents", 2025‑2026) maps this hierarchy to address LLM context limits, long‑term interaction loss and low retrieval efficiency.

L1 – Working Memory

L1 trims dialogue history to fit the token window while preserving essential information. Key design rules:

Separate SystemMessage : always keep, never prune.

Group tool calls so that AIMessage(tool_calls=[...]) + ToolMessage + ToolMessage remain together; splitting them causes API errors.

Iterate groups from newest to oldest, adding whole groups until the token budget is reached.

Inject L2 summaries as additional SystemMessage when available.

AIMessage(tool_calls=[...]) + ToolMessage + ToolMessage  →  one inseparable group
HumanMessage                                 →  independent group
AIMessage(no tool_calls)                     →  independent group

L2 – Episodic (Context) Memory

L2 applies progressive summarisation to keep token usage constant across many dialogue rounds. After every N rounds (commonly 10), the new segment is summarised by an LLM and merged with the previous summary.

Round 10: messages[0:10] → LLM → summary v1
Round 20: messages[10:20] + summary v1 → LLM → summary v2
Round 30: messages[20:30] + summary v2 → LLM → summary v3

The workflow stores the end index of each snapshot to avoid re‑processing the whole history:

1st summary (sort_order=10):
  snapshot_messages[0:] → LLM → summary_v1 → persist(end=len)
2nd summary (sort_order=20):
  range_end = previous end
  snapshot_messages[range_end:] → LLM (with existing_summary) → summary_v2 → persist(new end)

Prompt templates used:

Generate a concise Chinese summary of the following dialogue, preserving key topics, decisions and contextual information.

Merge the existing summary with new dialogue into a more concise Chinese summary, removing redundancy.

L3 – Semantic (Long‑Term) Memory

L3 extracts facts, preferences, habits and long‑term instructions from completed sessions and stores them for semantic retrieval.

Example extracted facts (initial confidence 0.8):

Preference: user prefers concise answers.

Fact: user is a Python backend developer.

Instruction: always include code examples in answers.

Semantic deduplication and conflict resolution:

If a new fact’s embedding similarity > 0.92 to an existing one, boost the existing confidence by +0.05 instead of adding a duplicate.

If similarity is between 0.7 and 0.92 and the fact types match, an LLM decides whether the facts conflict. If a conflict is detected, the old fact is marked "superseded" and the new fact receives a higher version number.

Storage pipeline:

store_facts(facts):
  1. filter low‑confidence entries
  2. capacity control – evict low‑value memories when over limit
  3. batch embed (MemoryEmbeddingService.embed_batch)
  4. for each fact → _smart_store():
     a. find_similar_memories (top 3 vectors)
     b. if sim > 0.92 → dedup & boost confidence (+0.05)
     c. if 0.7 < sim < 0.92 and same type → _detect_conflict (LLM)
        → if YES → _resolve_conflict (increment version, supersede old)
     d. if no match → insert new record with embedding

Retrieval at query time embeds the user question, performs a semantic search, formats the top results and injects them into the system prompt. If embeddings are unavailable, n‑gram keyword matching is used.

Memory decay runs periodically via decay_memories and removes stale or low‑confidence entries based on two rules:

Expired & low‑frequency: created > max_age_days and access_count ≤ 2.

Low confidence: confidence < decay_min_confidence (e.g., 0.3), often triggered by repeated negative feedback or newer contradictory facts.

Current L3 implementation is a flat "append‑only" fact list; retrieval returns isolated fragments and cannot yet answer questions such as which of two conflicting preferences is the latest, whether overlapping facts should be merged, or how to correct a remembered preference.

Final Summary

The system adopts a progressive‑enhancement design: the L1‑L3 three‑level memory framework forms the core, and a five‑stage enhancement pipeline can be toggled independently, ensuring zero‑intrusion and graceful degradation.