How a 1.5B Parameter Model Can Add External Knowledge to Any Frozen LLM

Background and Motivation

Large language models stop learning after pre‑training, so adding new knowledge requires one of three common approaches—retrieval‑augmented generation (RAG), fine‑tuning, or implicit memory—each with its own drawbacks such as rising inference cost, catastrophic forgetting, or representation coupling.

MEMO Core Idea

MEMO separates knowledge storage from reasoning. An Executive (a frozen, possibly closed‑source LLM such as a 32B model) handles reasoning, while a Memory model (a 1.5B‑parameter model fine‑tuned on a specific corpus) stores the details. The two models never share weights; the executive queries the memory model whenever it needs information.

Formal Framework

Let M_θ be the frozen LLM and D = {d_1,…,d_N} the target corpus. Knowledge integration is defined by a pair (Φ, f) where Φ maps the corpus to a representation K = Φ(D) and f combines K with the executive to answer a query q as f(M_θ, K, q). MEMO instantiates K as the parameters φ of a small model M_φ (with φ ≪ θ) trained on D.

Stage 1: Building the Memory Model

MEMO uses a five‑step data‑synthesis pipeline driven by a generator model M_{gen}:

Input: corpus D, generator M_{gen}, document groups G = {G_1,…,G_R}
Q_final ← ∅
for each document d in D:
    C ← Chunk(d)                     # split into text blocks
    Q_ver_d ← ∅
    for each chunk c in C:
        Q_dir, Q_indir ← M_{gen}(c)          # step 1: extract explicit & implicit facts
        Q_raw ← Q_dir ∪ Q_indir
        Q_mrg ← M_{gen}(Q_raw)              # step 2: merge related QA pairs
        Q_con ← Q_raw ∪ Q_mrg
        Q_ver ← M_{gen}(Q_con, c)            # step 3: verify & rewrite for self‑consistency
        Q_ver_d ← Q_ver_d ∪ Q_ver
    Q_ent_d ← M_{gen}(Q_ver_d)               # step 4: generate entity‑traceability QA
    Q_final ← Q_final ∪ Q_ver_d ∪ Q_ent_d
for each group G_i in G:
    Q_cross ← M_{gen}(⋃_{d∈G_i}(Q_ver_d ∪ Q_ent_d))   # step 5: cross‑document synthesis
    Q_final ← Q_final ∪ Q_cross
return Q_final

Each step is explained in the paper: step 1 extracts facts, step 2 merges them into multi‑fact questions, step 3 checks self‑consistency and rewrites ambiguous pairs, step 4 creates provenance questions to address the “reversal curse”, and step 5 discovers cross‑document links that are hard for retrieval systems.

After obtaining the reflection set Q_{final}, the memory model is trained with a full‑sequence‑fine‑tuning loss:

L(φ) = - \sum_{(q_i,a_i) \in Q_{final}} \sum_{t=1}^{|a_i|} \log M_{φ}\bigl(a_i^{(t)} \mid q_i, a_i^{(1\!:\!t-1)}\bigr)

During training the memory model never sees the source documents, forcing it to internalize knowledge in its weights.

Stage 2: Structured Inference Protocol

The executive follows a three‑stage protocol for each query q:

Grounding : decompose q into atomic sub‑questions {q'_1,…,q'_J}; the memory model answers each independently, producing {m_1,…,m_J}. (Budget = 1 round)

Entity Identification : iteratively ask the memory model to narrow down candidate entities until a single entity e* is identified or the budget (7 rounds) is exhausted.

Answer Seeking & Synthesis : with e* as a condition, collect supporting facts m_{seek} and synthesize the final answer<br/> â = M_θ( q , {m_j}_{j=1..J} , e* , m_{seek} ). (Budget = 8 rounds)

The protocol keeps each exchange short, making inference cost independent of corpus size and fully compatible with black‑box LLM APIs.

Empirical Findings

Ablation experiments on the BrowseComp‑Plus benchmark show that the structured protocol raises accuracy from ~32.6 % (single‑round) to 54.2 % (full three‑stage), while merely increasing the number of rounds without structure plateaus. Plug‑and‑play tests swapping the executive (e.g., Qwen2.5‑32B → Gemini‑3‑Flash) improve scores by +12.5 % to +26.7 % without retraining the memory model.

MEMO is markedly robust to noisy documents: retrieval‑based baselines drop 5–11 % points when distractor texts are added, whereas MEMO’s performance changes by at most ±1.8 %.

Continual Update via Model Merging

For new corpora D_i, a separate memory model M_{φ_i} is fine‑tuned from a shared base φ_0. The task vector τ_i = φ_i - φ_0 captures the shift caused by D_i. Models are merged as<br/> φ_{merged} = φ_0 + \sum_i λ_i τ_i (linear) or with more sophisticated schemes such as TIES. Linear merging reduces compute from Θ(K²) (full retraining) to Θ(K) and avoids catastrophic forgetting, though it incurs a modest accuracy loss (11–19 % points on some benchmarks).

Limitations

Fixed memory‑model capacity limits the amount of compressible knowledge.

Up‑front cost: the synthesis pipeline must run before any query.

Step 5’s quadratic complexity O(k·C²·Q²) makes long documents expensive.

Domain sensitivity: some pipeline steps that help Wikipedia‑style data hurt narrative texts.

Over‑fitting and vocabulary redundancy appear after the second epoch.

Reduced traceability: answers no longer expose the original source document, raising audit concerns.

Future Research Directions

Accelerate the pipeline, especially the cross‑document synthesis step.

Study how memory‑model size should scale with corpus size and executive capability.

Develop more accurate merging techniques that approach full‑retraining performance.

Explore reinforcement‑learning objectives for memory‑model training to mitigate over‑fitting.

Design architecture‑aware adapters (e.g., LoRA variants) that respect parameter distribution.

Optimize prompt budgets and enable dynamic, streaming knowledge updates.

Build provenance mechanisms to restore answer traceability.

Conclusion

MEMO demonstrates that separating knowledge storage from reasoning—by attaching a compact, fine‑tuned memory model to any frozen LLM—yields a system that combines the durability of fine‑tuning, the replaceability of RAG, and the compactness of implicit memory, while remaining robust to retrieval noise. It does not replace retrieval for all tasks, but offers an elegant re‑framing for integrating external knowledge into black‑box LLMs.