Cortex: Turning Agent Experience into Long-Term Parametric Memory

Problem: Agents Need Long-Term Parametric Memory

When an agent completes a task using context (documents, tool APIs, corrected facts), the experience remains only in the current context window. Standard in-context learning does not update model parameters, so the agent does not automatically improve on future tasks. Recursive Self-Improvement (RSI) requires a mechanism to turn transient context into lasting parametric memory.

Inspired by biological memory consolidation, researchers from Zhejiang University, ByteDance Seed, and Tsinghua University introduce Cortex , a general experience consolidation method. It converts the benefit provided by context into permanent parameter changes while preserving existing capabilities. Cortex can also incrementally expand parameter capacity; new components merge into the original network layers. The team reports that consolidating agent experience on ALFWorld with a 120B model takes about 30 minutes on 8 A100 GPUs. Cortex-Structural has been integrated into Hugging Face's PEFT library ( https://huggingface.co/docs/peft/index), supporting both dense and MoE models with standard fine-tuning and document consolidation examples.

Formalizing Memory Consolidation (MC)

The work uses Context (C) as the carrier of experience — tool docs, interaction summaries, corrected facts, or behavioral guidance. The context gap is the performance difference between the model with and without C, providing an observable measure of what the experience teaches.

Two query sets define the objective:

Q⁺ : checks whether the model learned the intended change (e.g., using a new API, answering with new facts).

Q⁻ : checks whether unrelated capabilities are retained (e.g., coding ability, language fluency).

Memory Consolidation aims to find an update such that, without accessing the original context C, the model recovers the target gain on Q⁺ while preserving performance on Q⁻. Two metrics quantify success:

Consolidation Rate (CR) = (gain of updated model without C on Q⁺) / (gain of original model with C on Q⁺).

Retention Rate (RR) = (performance of updated model on Q⁻) / (performance of original model on Q⁻).

Both metrics are relative to the original model without C on Q⁺. CR = 100% means full recovery of the measured context benefit; RR = 100% means parity on retention tasks. Values can exceed 100%. The definition specifies the goal but not the training algorithm; Q⁺/Q⁻ are held-out evaluation sets.

Cortex: Three-Stage Consolidation Pipeline

2.1 What to Consolidate and How: Prospective Rehearsal + Distillation

The same context contains many details; what matters depends on future use cases. Cortex first organizes learning material around the question: In which future situations will this experience help, and how should the model's answers or actions change? This step, called prospective rehearsal , uses the pre-update model to generate likely questions from the context and answer them while seeing the context. For example, a data-processing tool doc becomes exercises like filtering records, handling missing fields, or fixing call errors. The generated (question, answer) pairs link the experience to concrete usage scenarios.

For agent tasks, questions come from the training environment and successful offline trajectories are kept as demonstrations; no further environment interaction occurs during consolidation.

Next, Cortex adopts context distillation : a frozen pre-update model acts as teacher (receives context C), while the student (same architecture, no context) learns to mimic the teacher's behavior on the rehearsed questions. Training aligns both output distributions (KL divergence) and intermediate FFN residual representations (MSE). A small set of general capability samples is mixed in to limit unrelated changes.

2.2 Where to Write: FFN-Focused Parameter Updates

Cortex restricts updates to the Transformer's feed-forward networks (FFNs), which prior work interprets as key-value memories. Experiments confirm that updating only FFNs yields better learning-retention trade-offs than updating attention as well, with fewer trainable parameters.

Two writing schemes are designed, drawing analogy to synaptic plasticity:

Cortex-Synaptic : freezes original weights and applies low-rank incremental updates to existing projections (similar to LoRA).

Cortex-Structural : keeps original weight blocks and adds new nonlinear feature channels, providing extra representational capacity for new experience.

For MoE models, Structural uses a shared expansion branch — a single set of new parameters that every token passes through — ensuring the added capacity receives uniform training. Router weights stay frozen. Both writing styles can be merged back into standard layers (low-rank into existing matrices, structural into wider FFNs), so inference requires no special retrieval of the original context.

2.3 Continuous Consolidation: Learn New, Rehearse Old

Each round freezes the base model and all previous writes, training only the newly added parameters. However, frozen parameters do not guarantee frozen behavior: new parameters participate in computation and can alter answers to old questions. Cortex therefore retains the previously generated practice questions. Before each update, the current model re-answers those questions to produce supervision signals for rehearsal — without re-reading the original contexts. In continuous fact-updating experiments, the pre-update model supervises rehearsal of accumulated experience, while the initial model supervises general capability retention. Historical questions are used only for training replay, not for inference-time retrieval.

Memory Consolidation Testbed

A unified benchmark evaluates five experience types with consistent context input format and Q⁺/Q⁻ criteria:

Behavior guidance (steering)

Tool documentation learning (API usage)

Fact updating (knowledge insertion)

Knowledge suppression (unlearning)

Agent tasks (ALFWorld, τ²-Bench)

Models range from 4B to 120B parameters, including dense and MoE, instruction-tuned and reasoning variants. Baselines span four categories: task-specific methods (knowledge editing: ROME, MEMIT, AlphaEdit, WISE; unlearning: GA, RT, DPO), steering (CAA, RePS, SPLIT, plus MemP for agents), test-time training/online memory (LoCAS, δ-Mem), distillation-based (OPCD, SELF-PARAM, SDCD), and hypernetworks (Doc2LoRA, SHINE). Each method is evaluated on its applicable tasks; an ablation also fixes the training data (using Cortex's rehearsal set) to isolate the effect of learning algorithm and write location.

All comparisons center on three states: original model without C, original model with C, updated model without C. The first two establish the context gap; the third tests consolidation. Retention evaluation measures the cost of learning.

Experimental Results

4.1 Main Experiments: Diverse Experiences Consolidated Successfully

Across 15 CR–RR comparisons, at least one Cortex variant achieves the best learning-retention trade-off among methods that report both metrics and do not use original context at inference.

Behavior guidance : ~95–100% CR, ~99–100% RR (language fluency).

Batch fact updating on three Qwen models: 80–91% CR, far above the best editing/fine-tuning baselines (36–46%).

Tool documentation : Qwen2.5-14B Structural lifts average pass@1 from 25.3% to 57.4% (73.7% CR, 97.6% RR), outperforming OPCD, SELF-PARAM, SDCD.

Agent tasks : GPT-OSS-120B on ALFWorld jumps from 55.2% (no context) to 97.8% after Structural consolidation, slightly above the 97.3% with context provided (101.2% CR, 95.2% RR). Qwen2.5 also exceeds 100% CR. However, τ²-Bench shows incomplete recovery or even degradation.

Key observations:

Consolidated performance can surpass the original with-context upper bound, but this is task-dependent.

Hypernetworks (Doc2LoRA, SHINE) excel at single-shot updates (Doc2LoRA ~95% CR on single fact) but collapse in batch settings (Doc2LoRA ~24% CR on batch facts; SHINE target performance below original model on batch API learning). SHINE achieves ~100% CR on unlearning but only 4–10% RR.

TTT methods (LoCAS, δ-Mem) preserve old capabilities (RR ~98–101%) but learn little new (CR −9% to 11% on batch API; LoCAS 27%, δ-Mem ~0% on batch facts).

Writing style matters: Structural leads in 6/8 code settings; Synaptic leads in 7/8 fact-editing settings. Retention gaps can be large (GPT-OSS-20B ALFWorld: Structural RR ~95%, Synaptic ~23%).

Batch consolidation difficulty is not predicted by context length alone (e.g., ~599-token suppression context harder than ~5,152-token API context on Qwen2.5).

Directed suppression struggles with selectivity: GPT-OSS-20B achieves 97–100% CR but only 76–83% RR; adjacent non-target knowledge degrades more than general reasoning.

4.2 Continuous Learning: 50 Sequential Fact Updates

On Qwen2.5-14B-Instruct, 50 facts learned sequentially (three seeds). Structural reaches 88.9% CR, 99.8% RR; Synaptic 80.6% CR, 97.2% RR; best baseline ROME only 28.5% CR. Structural adds ~0.005% of original parameters per round, ~0.25% cumulative after 50 rounds.

Final advantage stems from both strong initial learning and minimal forgetting: Structural's average CR over all 50 facts drops only from 91.3% to 88.9% (2.4 pp), while WISE falls from 52.0% to 28.0%. The earliest 10 facts retain 97.2% average CR after 40 subsequent updates.

However, remembering facts ≠ flexible use: direct fact QA ~95% for both, but compositional reasoning yields 49.4% (Structural) vs 30.7% (Synaptic). Structural shows better compositional ability from the start and degrades less. Output stability also improves: Structural 0% repetition/length-limit terminations, Synaptic 0.8%, baselines 86–91%.

4.3 Ablation Studies: Source of Gains

Learning algorithm matters : On identical rehearsal data, distillation (matching teacher output distributions) beats imitation (matching generated answers) by 16.6 pp CR on GPT-OSS-120B batch API. Even with same distillation, LoRA underperforms Cortex-Structural by 23 pp on Qwen2.5-14B.

FFN-only updates suffice : Fixing LoRA rank, updating only FFN outperforms updating FFN+attention by 3.8 pp CR and 4.0 pp RR with fewer trainable parameters.

Rehearsal is essential despite frozen weights : Without historical rehearsal, earliest 25 facts drop to 40.5% (Structural) and 8.0% (Synaptic) accuracy after 25 more updates; with rehearsal both exceed 96%. Saving only questions (regenerating answers each round) matches saving full Q&A pairs (CR difference <1 pp).

Intermediate representation alignment aids retention : Adding MSE on FFN residuals improves RR by 2.7 pp (Synaptic) but reduces CR by 4.0 pp; for Structural both metrics improve. Aligning internal processing helps preserve behavior but may constrain new learning.

MoE shared expansion trains added capacity more uniformly : With equal total new parameters, shared branch gains 4.6 pp (ALFWorld) and 10.3 pp (API) CR. With equal per-token new parameters, shared needs 1/8 the parameters for similar CR. Training logs show shared channels receive uniform updates; many independent expert branches see almost no data.

Small capacity is sufficient : Structural at 0.005% per round already achieves 88.9% CR; increasing capacity does not consistently improve results under this training regime.

Discussion

5.1 What Experience Deserves Consolidation

Experience that aids future decisions — new knowledge enabling previously impossible answers, reusable task strategies, correction of errors or outdated info — should be consolidated. Ephemeral details (e.g., a temporary filename in a single API call) lack reuse value. Volume of logs ≠ amount of learning.

5.2 Why Parametric Memory: Direct Participation in Reasoning

External retrieval memory requires the relevant experience to be found before it can influence reasoning. Some connections only emerge during reasoning when combined with other knowledge (Kekulé's benzene insight from a snake biting its tail — "snake" and "benzene" are lexically distant but structurally linked). If the experience isn't retrieved, it cannot participate. Parametric memory integrates experience into the model's own computation, allowing it to interact with existing knowledge during every forward pass.

5.3 Three Ideals for Parametric Memory

Distributed integration : Memory spans all layers, enabling new experience to join cross-layer knowledge circuits (Yao et al., 2024). Cortex writes to FFNs at every layer.

Growable capacity : Model can expand to accommodate accumulating knowledge. Cortex-Structural adds feature channels for this purpose.

Native access : Consolidated memory runs as part of the standard forward pass, no external retrieval or module selection needed. Cortex's writes merge into standard FFN/MoE layers.

5.4 Distinction from Related Work

MC unifies knowledge editing, skill learning, steering, and unlearning under a single consolidation framework with shared CR/RR evaluation. Unlike continual learning (given training set) or classic context distillation (given context + input distribution), MC provides only the model and context(s); the method must construct training data (Cortex's prospective rehearsal does this). Cortex combines distillation, parameter writing, and rehearsal into one implementation. Hypernetworks pre-train a generator; Cortex trains per experience. Methods like WISE/GRACE require selecting which memory module to use at inference; Cortex accumulates all writes into one model, enabling multiple past experiences to jointly participate in a single reasoning episode (e.g., tool usage + troubleshooting strategy learned at different times).

Paper: https://arxiv.org/abs/2610.20399 | Code:

https://github.com/zjunlp/cortex