Becoming a Memory Engineer: Lessons from Stanford, Microsoft, Anthropic, and NVIDIA

Step 1 – Re‑frame memory as a metabolic system

Typical pipeline loads a vector DB, ingests history, retrieves top‑k. When history exceeds the context window, write‑path cost dominates query latency and stale states accumulate. Memory must be treated as a system with explicit cost and lifecycle.

Step 2 – Four observation angles

Stanford: cost of memory

Microsoft: what to retain

Anthropic: who controls retention

NVIDIA: hardware impact

Step 3 – Price the write path (Stanford)

Stanford’s real‑world study shows the dominant bill comes from the build phase (LLM pre‑fill + embedding), which is paid once and invisible to users. Query phase (retrieval + generation) is observed per request. Maintenance (deduplication, compression, forgetting) is usually missing. Energy spent on building exceeds that of 300 subsequent queries on the same memory.

Step 4 – Measure energy per correct answer

Two systems with identical accuracy can differ by up to 47 × in energy per correct answer. Reporting both quality and per‑answer cost is recommended.

Step 5 – Choose a memory family (Stanford)

Memory is classified into four families: raw context, flat retrieval, structured extraction, fully‑agentic. No family dominates across build cost, query speed, and accuracy; practitioners must consciously select the cost they are willing to incur.

Step 6 – Store facts and skills instead of raw logs (Microsoft PlugMem)

Raw log example: “5 May, user said: I always publish via GitHub Actions, never manual.” → discard Fact: User always publishes via GitHub Actions. Skill: When a deployment fails, first check Actions before touching production.

Step 7 – Let the model manage its own context (Microsoft Memento)

Memento pushes high‑density notes into the model and deletes the original reasoning. Peak memory drops 2–3 × and throughput nearly doubles. The skill is learned via ordinary fine‑tuning. Re‑building context from notes loses ~15 percentage points in accuracy, showing that forgetting is not deletion and remembering is not mere storage.

Step 8 – Keep control over retained content (Anthropic)

Memory is stored as ordinary files, allowing agents to read/write with familiar tools. Files provide exportability, auditability, and programmatic control.

Step 9 – Store memory in deletable files (Anthropic)

Any storage that cannot be opened or edited is automatically out of the operator’s control.

Step 10 – Scope, audit, rollback

/memory</code>
<code>/org read‑only conventions.md, past‑incidents.md</code>
<code>/user‑4821 read‑write preferences.md, skills/</code>
<code>audit.log which‑agent, which‑session, what‑changed, when</code>
<code>→ export, rollback, or erase any memory

Teams using this approach cut first‑day errors by 97 % and improve verification speed by ~33 %.

Step 11 – Manual validation before automation (NVIDIA)

Run a manual pass: feed the agent real history, ask it to extract facts/skills, flag contradictions, and compute the cost of keeping the output fresh.

Step 12 – Treat building as a background task (NVIDIA)

Building behaves like an offline indexing job; large writes can block the scheduler. Strategies: rate‑limiting, batching, or deferring builds. Memory decisions should be measured in KV‑cache usage on HBM. Full‑context cost grows quadratically; KV‑cache fills HBM and is evicted between sessions, forcing re‑payment.

vLLM MEMENTO on B200: 4,290 tokens/s vs. 2,447 tokens/s; same batch time 693 s vs. 1,096 s.

Step 13 – Order of delivery

Build the write path, store facts and skills, let it warm up for weeks.

Manually run a few contradiction checks; only schedule automation when unexpected conflicts appear.

Introduce forgetting and maintenance before storage balloons.

Finally, tune the hardware layer—batch builds, limit retrieval, and monitor KV‑cache once scale is real.

Conclusion

The core challenge is not forgetting but the lack of conscious forgetting. A memory engineer decides what to discard, what to distill, and how to prune before growth collapses the system. Following the fifteen steps moves from naive storage to engineered, cost‑aware, and hardware‑aligned agent memory.

Sources: Stanford “Agent Memory: Characterization and System Implications”; Microsoft Research PlugMem and Memento; Anthropic “Built‑in Memory for Claude Managed Agents”; NVIDIA hardware analysis (vLLM, H100, B200).