Memory‑Based Self‑Evolution: Redefining LLM Agents Beyond Parameter Updates

Problem Statement

In the second stage of large‑model deployment, relying solely on Supervised Fine‑Tuning (SFT) and Reinforcement Learning (RL) incurs high computational cost, suffers from slow knowledge updates, and is prone to catastrophic forgetting. To address these limitations, a new agent‑optimization paradigm called Memory‑Based Self‑Evolution is proposed.

Core Idea: Dynamic Memory System

The paradigm introduces a Dynamic Memory System that continuously records interaction trajectories (actions, feedback, states) into an experience bank . During inference the agent retrieves relevant memories and injects them into the current context, enabling implicit reasoning, error correction, and continual improvement without modifying the base model parameters.

Four Representative Memory‑Based Solutions

Dynamic Cheatsheet (Test‑Time Learning with Adaptive Memory)

A Memory Curator evaluates the output of the generator, filters low‑quality information, and updates a concise “cheatsheet” that is consulted in subsequent queries.

Provides lightweight, on‑the‑fly correction compared with full fine‑tuning or static Retrieval‑Augmented Generation (RAG).

ReasoningBank (Scaling Agent Self‑Evolution with Reasoning Memory)

Generates multiple reasoning paths for the same query (parallel scaling) and extracts high‑consistency patterns (chain‑of‑thought scaling) to form a collective “wisdom bank”.

Key technique: MaTTS (Memory‑aware Test‑time Scaling) that aggregates successful and failed trajectories into stable knowledge.

Agentic Context Engineering (ACE) (Evolving Contexts for Self‑Improving Language Models)

Transforms business SOPs into structured playbooks containing strategies, hard rules, code snippets, and troubleshooting guides.

Combines offline prompt optimization with online test‑time updates via three components:

MemGen (Weaving Generative Latent Memory)

Injects latent tokens into the LLM’s hidden state via a dual‑LoRA adapter, creating a latent memory that is stored directly in model parameters (weights W).

The Memory Trigger captures the current hidden state and decides whether to awaken the latent memory; the Weaver concatenates generated latent tokens to the hidden state, allowing the agent to recall knowledge as intuitively as human intuition.

Vertical Application: Insurance Industry

Insurance requires rigorous legal, medical, and financial knowledge. Pure LLM knowledge leads to hallucinations and insufficient precision. By integrating external knowledge bases (RAG) with the self‑evolution mechanisms above, each claim review or policy interpretation becomes a knowledge‑capture event, gradually forming a “knowledge flywheel” that continuously improves accuracy and processing speed.

Future Outlook

Transitioning from static parameter updates to lifelong‑learning agents bridges the gap between generic models and domain‑specific expertise. Agents equipped with memory‑based self‑evolution can evolve from mere tools into collaborative experts that grow with every interaction.

References

Dynamic Cheatsheet: Test‑Time Learning with Adaptive Memory, arXiv:2504.07952 (2025).

ReasoningBank: Scaling Agent Self‑Evolving with Reasoning Memory, arXiv:2509.25140 (2025).

Agentic Context Engineering: Evolving Contexts for Self‑Improving Language Models, arXiv:2510.04618 (2025).

MemGen: Weaving Generative Latent Memory for Self‑Evolving Agents, arXiv:2509.24704 (2025).