Repositioning the Three Architectural Axes of LLM Memory

Introduction

The recent survey Memory for Large Language Models (arXiv:2607.25380) provides a systematic taxonomy for LLM memory mechanisms. It groups dozens of approaches into a common design space and reduces them to three fundamental questions: where is the information stored, when is it updated, and how long does it influence the model.

Three Orthogonal Axes

Representation : implicit (tied to the computation flow) vs. explicit (independent storage with read/write interfaces).

Update dynamics : offline (updated only during training) vs. online (updated during inference).

Persistence : short‑term (affects the current token/sequence) vs. long‑term (remains across sessions).

These axes are not interchangeable; for example, a Mixture‑of‑Experts (MoE) module can be an offline‑updated explicit memory, while sliding‑window attention is an online short‑term memory.

Memory Mechanisms

The survey classifies LLM memory into four technical families:

Attention : a differentiable, content‑addressable work‑set that flexibly accesses recent tokens but incurs cost proportional to context size.

Recurrent state : linear‑time or state‑space models that compress history into a fixed‑size hidden state, enabling cheap computation but making precise recall harder.

Fast parameters (e.g., Titans, Test‑Time Training): small parameter subsets that adapt during inference, trading stability for rapid adaptation.

Lookup storage (kNN‑LM, Engram): external, addressable entries that can be retrieved and edited independently of the forward pass.

Engineering Mapping

In practice, an agent’s memory should be split into four categories, each with its own address, update policy, and lifecycle:

Work‑set : current logs, code diffs, tool outputs – short‑lived, fast to read/write.

Compressed state : summarized task status – fixed capacity, may lose fine‑grained details.

Long‑term items : confirmed project constraints or historical decisions – versioned and auditable.

Raw evidence : original logs, test reports, approvals – immutable source of truth.

Addressing, versioning, budgeting, verification, and rollback must be explicit in the architecture, similar to how databases handle tenant, user, project, and session scopes:

tenant / user / project / session

Evaluation Dimensions

The survey breaks down memory evaluation into seven orthogonal metrics:

Capacity – how much information can be stored before degradation.

Fidelity – deviation between retrieved and original content.

Persistence – length of influence across tokens or calls.

Robustness – stability under similar, conflicting, or irrelevant inputs.

Efficiency – compute, memory, latency, and update cost.

Structured dependency – ability to compose dispersed facts for reasoning.

Forgetting & stability – controllable decay of old knowledge.

Long‑context benchmarks alone cannot isolate memory improvements; each dimension should be measured separately.

Implementation Guidelines

A practical rollout can start with a minimal closed‑loop:

original evidence → candidate memory → scope & time check → publish or reject → traceable recall

Example metadata for a memory entry:

scope: project/release
source: ci-run-1842
observed_at: 2026-08-09T10:30:00+08:00
status: candidate
expires_at: 2026-09-09T00:00:00+08:00
evidence_ref: logs/ci-run-1842
supersedes: null

Key checks include admission signals, update strategy, eviction policy, and consolidation criteria. Tests should verify provenance, conflict handling, rollback auditability, graceful degradation when the memory service is unavailable, and policy enforcement for side‑effects.

Conclusion

No single mechanism dominates; attention, recurrent state, fast parameters, and lookup storage each consume part of the system budget and introduce distinct failure modes. Mixed architectures are likely to remain the norm, and engineers should first separate work‑set, task state, long‑term memory, and raw evidence before attempting a unified memory platform.