Designing AI Agent Memory: Evolving from Context Stitching to a Self‑Optimizing Cognitive System (Part 2)

01 Vector Retrieval – Making Memory Semantically Understandable

Traditional n‑gram + ILIKE search matches only literal strings, failing to retrieve semantically equivalent statements such as a user preference for Python when the wording differs. Embedding converts text into high‑dimensional vectors; cosine similarity measures semantic closeness, allowing "meaning‑based" matching.

HyDE (hypothetical‑answer generation) first lets the LLM produce a plausible answer, which is then used to query the memory store, narrowing the semantic gap between short queries and long memories. A second‑stage reranker (CrossEncoder or Cohere) re‑scores the initial candidates, filtering out false positives and delivering a few highly relevant results with modest compute cost.

Result: higher recall for L3 memories, reduced irrelevant context injected into L1 and L2, and lower token consumption.

02 Memory Lifecycle – Activating Memories

L3 should not be a static list; it needs a lifecycle similar to human memory: reinforcement of repeated facts, updating of contradictory information, and expiration of stale entries. The design implements three mechanisms:

Semantic deduplication: merge two memories when vector similarity > 0.92, preventing redundant "same meaning, different wording" entries.

Conflict resolution: for similarity between 0.7 and 0.92, invoke a lightweight LLM to answer YES/NO on contradiction, then apply version‑overwrite.

Confidence evolution: assign each memory a confidence score between 0 and 1; positive evidence (repetition, user acceptance) raises the score, negative evidence (user denial) lowers it, giving the system a dynamic weighting.

Process flow: vector similarity → conflict detection → confidence enhancement & version increment → decide insert, merge, or overwrite.

Compared with many market solutions that only deduplicate, this approach also handles preference changes (e.g., "Java" → "Go") and eliminates repeated or contradictory facts.

03 User Profile Engine – Building a Real User Model

LLMs generate better responses when supplied with a coherent user portrait rather than scattered memory fragments. Two‑level context injection is used:

Global profile (low‑frequency, high‑density): periodically summarize all active memories into a concise user summary (≤200 tokens) placed at the beginning of the system prompt.

Local relevant memory (high‑frequency, precise): for each turn, inject the most similar memories (≤300 tokens) based on vector similarity.

This resembles providing the LLM with a "user résumé" plus the "current conversation highlights," ensuring both global consistency and local relevance. The profile itself carries a version number and can be rebuilt incrementally.

请根据以下用户的历史记忆信息，生成一段简洁的用户画像描述（不超过 150 字）。
画像应包含：
1. 用户的职业/角色
2. 主要技术栈或领域
3. 沟通风格偏好
4. 当前关注的项目或主题
用户记忆：{memories}
用户画像：

04 Explicit & Implicit Feedback – From Passive Recording to Active Learning

Without feedback, a memory system only accumulates more data without improving accuracy. The design introduces a reinforcement‑learning loop that combines explicit UI feedback and implicit usage signals.

Explicit feedback presents the recalled memories to the user, who can:

👍 确认（confidence +0.1）
👎 否定（confidence -0.2）
✏️ 修正（更新 content，重算 embedding，confidence = 0.9）

Implicit feedback treats a memory as "satisfied" if it is recalled and not corrected in the next two dialogue turns, adding a small confidence boost (+0.02). An explicit correction during those turns subtracts 0.1.

These combined signals drive profile regeneration and enable the system to "learn from the user" continuously.

05 Governance & Self‑Healing – Keeping the Memory System Healthy

Long‑term operation requires governance to avoid storage blow‑up and quality decay. The proposed principles are:

Capacity control: default per‑user limit of 500 memories; excess items are evicted based on a value score = confidence × (access_count + 1).

Timed decay: a daily cron reduces confidence for items that are expired, low‑frequency, and have confidence < 0.3.

Operational monitoring: expose rich APIs and metrics so users can monitor and manage their memory stores.

These measures prevent performance degradation, stale or low‑quality memories polluting L1/L2 context, and ensure observable, self‑healing behavior.

06 Conclusion

By iterating through the five stages—vector retrieval, lifecycle management, profiling, feedback loops, and governance—the system transforms from a simple memory warehouse into a learning, reflective, self‑optimizing cognitive engine capable of adapting to evolving user preferences.