Large-Model Memory Panorama: The 3‑D Taxonomy Unveiled by Tsinghua’s Tang Jie Team

Background

Scaling laws (parameters, data, compute) have driven recent LLM breakthroughs, but memory is emerging as an equally critical dimension. Traditional LLM memory is implicit—by‑products of forward computation such as the Transformer KV cache or hidden states of linear RNN/SSM—and grows quadratically with sequence length, limited by context windows.

Extending context windows alone is insufficient. Researchers are introducing explicit, addressable, and persistent memory modules that can be updated during inference.

Three‑Dimensional Taxonomy

Representation – Implicit vs. Explicit

Update Dynamics – Offline vs. Online

Persistence – Short‑Term vs. Long‑Term

This taxonomy unifies scattered frontier works and guides the design of more efficient, long‑lasting LLMs.

Representation

Implicit Memory : tightly bound to forward computation, lacking independent read/write interfaces (e.g., attention KV cache, RNN/SSM hidden states).

Explicit Memory : independent storage with clear read/write semantics, not confined to the forward pass (e.g., external datastore, test‑time updatable parameters, memory slots).

Update Dynamics

Offline : updated only during training via gradient descent (e.g., pre‑training parameters, traditional MoE).

Online : supports dynamic updates at inference time, allowing models to absorb new knowledge without full fine‑tuning (e.g., Titans, Test‑Time Training).

Persistence

Short‑Term : disappears with the end of a local context window or inference session (e.g., attention).

Long‑Term : information persists across contexts and even separate sessions.

Implicit Memory

Attention Mechanism : content‑addressed, pointwise working memory. Sparse attention or sliding‑window variants (e.g., StreamingLLM) reduce cost but capacity remains limited by window size.

Recurrent Sequence Memory : models such as Mamba (SSM), RWKV, Gated DeltaNet compress history into hidden states. Recent work (e.g., Kimi Delta Attention) upgrades scalar decay to channel‑wise dynamic state editing.

Limitations : capacity bounded by hidden dimension and window length; no independent controllable read/write semantics.

Explicit Memory

Parametric Memory Modules : Titans and TTT‑E2E embed memory in dedicated parameter blocks, enabling high‑frequency test‑time updates and decoupling context storage from the knowledge backbone.

Lookup‑Based Memory : Engram, kNN‑LM store information in non‑parametric slots or databases and retrieve via sparse routing during inference, dramatically expanding capacity.

Conditional Parameter Memory (MoE) : Mixtral‑type MoE architectures act as explicit memory by routing queries to context‑dependent parameter subsets.

Hybrid Architectures and System‑Level Challenges

No single memory paradigm solves all problems. Attention offers high recall but consumes large GPU memory; SSM is compute‑efficient but may lose fine‑grained detail; parametric memory adapts well but risks drift.

Hybrid memory architectures are becoming mainstream. Representative examples include:

Kimi Linear interleaving Kimi Delta Attention (≈3:1) with Multi‑head Latent Attention.

Qwen3‑Next mixing Gated DeltaNet with full‑attention layers.

Jamba and Samba combining attention and SSM layers.

Emerging directions focus on adaptive memory routing (e.g., AMOR decides when to invoke costly attention based on uncertainty; HAM routes tokens to a high‑fidelity KV cache or to a compressed recurrent state based on prediction error).

Key challenges remain:

What to write and when : metrics such as surprise, entropy, or prediction error may not reflect long‑term value.

Stability vs. Plasticity : test‑time updates can cause memory drift, interference, and error accumulation.

Capacity, Precision, and System Cost : high‑fidelity KV caches consume GPU memory; fixed‑size recurrent states may over‑compress history. Techniques such as PagedAttention, KV‑cache quantization, and memory integration must be co‑designed with model architecture.

Measuring “good memory” : long context length ≠ effective memory. Benchmarks such as RULER and LongBench assess persistence, interference resistance, update consistency, and efficiency.

Future Directions

Unified Memory Theory : develop a single framework that quantifies attention, SSM, and retrieval mechanisms via information compression and state transition.

Lifelong Parametric Memory : overcome the short‑term nature of test‑time training to achieve stable, incremental learning without full fine‑tuning.

Adaptive Memory Allocation : replace static designs with learnable controllers that decide storage medium (cache vs. state) and persistence level.

Algorithm‑Hardware Co‑Design : as context windows reach millions of tokens, memory read/write must be tightly coupled with GPU memory hierarchies.

Paper link: https://arxiv.org/abs/2607.25380