8×8 Matrix Gives LLMs Long‑Dialogue Memory with Just 0.12% Extra Parameters (δ‑mem)

δ‑mem introduces a compact 8×8 online state matrix that, without expanding context windows or altering the Transformer backbone, provides effective long‑term memory for large language models, achieving up to 1.31× performance gains on memory‑intensive tasks while adding only 0.12% parameters.

LLM memoryTransformerdelta-mem

0 likes · 15 min read

8×8 Matrix Gives LLMs Long‑Dialogue Memory with Just 0.12% Extra Parameters (δ‑mem)