Is RAG Doomed? Exploring Paths to True AI Memory and Continuous Learning

Are Memory and Retrieval Inherently Conflicting?

Current AI discussions focus on model memory and deep understanding. Expanding context windows and using Retrieval‑Augmented Generation (RAG) alleviate information access but keep memory external, forcing the model to re‑process the same data on every inference, which raises compute costs and hampers deep comprehension.

What Are RAG’s Core Limitations?

Engram co‑founders Dan Biderman and Jessy Lin argue that RAG suffers from three fundamental flaws:

Repeated reading wastes tokens and compute, similar to an employee searching yesterday’s notes every day.

RAG can only answer explicit queries and cannot perform proactive association or judgment.

The model lacks an intrinsic ability to decide *what* to look up, a shortcoming that becomes critical in complex tasks.

Insights from the Engram Interview

Engram, founded in 2025, raised $98 million in a Series‑A round to build a native continuous‑memory architecture that internalizes knowledge on private data instead of relying on external retrieval. Biderman, a former Stanford AI Lab post‑doc, and Lin, a former Meta FAIR researcher, stress that stuffing massive documents into prompts is merely externalized memory, not true learning.

They compare this to a human employee who, after repeatedly searching for the same information, wastes time and cannot develop an intuitive grasp of organizational context.

Human Brain as a Model for Continuous Learning

Biderman draws parallels with the brain’s lossy compression: we forget a hotel room number from a year ago but retain a home password because usage frequency and retrieval speed dictate what is internalized. During sleep, the brain consolidates experiences into long‑term memory, filtering noise—a mechanism Engram aims to emulate.

Engram deliberately avoids heavy heuristic rules for data selection, hoping the model can autonomously prioritize information, much like the brain distinguishes important from trivial experiences.

Open Questions and Future Directions

The boundary between internalized and retrieved knowledge remains unresolved. Biderman and Lin note that the criteria for what should be internalized vary across teams and scenarios, and no universal solution exists yet.

They envision a future system where external retrieval and internal memory cooperate: the model decides when to query and when to rely on its own consolidated knowledge, continuously refining this decision based on experience.