From Solving to Evolving: How RETROAGENT Gives AI Agents Real Retrospective Learning

Problem with Existing RL Agents

Current RL agents tend to be forgetful and overly conservative: they either converge to sub‑optimal policies or restart episodes without retaining information about past failures, because the standard training objective only rewards task completion.

RETROAGENT Framework

RETROAGENT introduces a retrospective dual intrinsic feedback mechanism that enables agents to evolve through self‑reflection after each episode.

Hindsight Self‑Reflection

After an episode the agent generates a “review report” and receives two forms of intrinsic feedback.

Intrinsic Numerical Feedback

The capability‑evolution reward assigns credit for partial progress (e.g., moving a box) even when the episode ends in failure, encouraging exploration of “failed‑but‑useful” trajectories.

Intrinsic Language Feedback

Successes and failures are distilled into natural‑language lessons that are stored in a Memory Buffer. Future episodes can query this buffer to retrieve relevant “mistake notes”.

Memory Retrieval: SimUtil‑UCB

To avoid over‑reliance on the most similar memories, RETROAGENT selects experiences using SimUtil‑UCB, which balances three factors:

Relevance : similarity to the current task.

Utility : historical effectiveness of the experience.

Exploration : an upper‑confidence‑bound bonus that promotes less‑used memories.

Empirical Evaluation

Experiments on four benchmarks—ALFWorld, WebShop, Sokoban, and MineSweeper—compare RETROAGENT against strong baselines GRPO and Qwen‑2.5‑7B. Reported improvements are:

ALFWorld: +18.3 % over GRPO.

WebShop: +15.4 %.

Sokoban: +27.1 %.

MineSweeper: +8.9 %.

Test‑time adaptation experiments show that, given multiple attempts, success rates approach 100 %.

Implementation Details

Dual‑variant design : an In‑Context version that uses prompts for reflection, and an RL‑trained version that learns a reflection policy with REINFORCE.

Memory retrieval balance : adding the UCB exploration bonus prevents collapse to a few old experiences.

Pairwise Induction : the model compares successful and failed trajectories to generate higher‑quality lessons.

Conclusion and Outlook

RETROAGENT shifts the agent objective from pure solving to continuous evolution, providing long‑term memory and self‑improvement capabilities. Future work includes multi‑task balancing of reflection versus decision goals and extensions to multi‑agent settings.

Paper: https://arxiv.org/pdf/2603.08561