How MemSifter Delivers High‑Precision, Low‑Cost Long‑Term Memory for LLMs

1. The LLM Memory Dilemma

When large language models (LLMs) engage in dozens of dialogue rounds, cross‑document research, or multi‑step agent tasks, they quickly exceed their context windows, leading to two core problems: (1) simple linear memory stores suffer from low retrieval precision, and (2) more complex enhanced retrieval pipelines incur prohibitive computational costs.

2. MemSifter Core Design

MemSifter solves this dilemma by completely decoupling memory retrieval from the main model inference. A lightweight, specially trained proxy model handles all heavy memory‑screening work, while the primary LLM receives only a highly distilled set of relevant snippets.

Think‑and‑Rank Mechanism

The proxy model follows a three‑step “Think‑and‑Rank” process:

Deeply decompose the current task to identify required key information.

Scan the entire interaction history and evaluate the relevance of each turn to the task.

Rank the turns by relevance and forward only the top‑K most pertinent segments to the main LLM.

This pipeline adds negligible overhead during indexing and only a tiny context window during inference, avoiding the heavy computation of traditional methods.

3. New Training Paradigm: Task‑Result‑Oriented RL

Instead of optimizing static relevance labels, MemSifter aligns memory quality directly with downstream task performance using a reinforcement‑learning (RL) framework that addresses two challenges: credit‑assignment ambiguity and lack of fine‑grained ranking supervision.

Marginal Utility Reward

The reward is computed by comparing task scores with and without retrieved memories, evaluating incremental gains at successive retrieval depths (Top‑1, Top‑2, …) and assigning credit only to memories that truly improve the final outcome.

Establish a no‑memory baseline score.

Incrementally add retrieved memories and measure task performance at each cutoff.

Reward the marginal improvement between successive cutoffs.

Rank‑Sensitive Reward

Inspired by DCG’s logarithmic decay, the reward weights decrease with lower ranking positions, ensuring that earlier retrieved information receives higher credit.

Training Optimizations

Warm‑start supervised training : use a small labeled set to teach the proxy basic output format and relevance judgment, mitigating cold‑start issues.

Dynamic curriculum learning : prioritize samples whose difficulty matches the model’s current capability, preventing over‑fitting to easy cases or collapse on overly hard ones.

Model averaging : after each training round, average the top‑K checkpoints on the validation set to smooth optimization and avoid performance spikes.

4. Experimental Results

MemSifter was evaluated on eight authoritative LLM memory benchmarks covering long‑dialogue recall, user‑profile modeling, multi‑hop reasoning, and deep research scenarios. It consistently outperformed five major baseline families (embedding retrieval, memory‑management frameworks, graph retrieval, generative re‑ranking, and native long‑context LLMs).

End‑to‑End Task Performance

Across all benchmarks, MemSifter achieved the best or near‑best F1 scores, e.g., 41.79 on LoCoMo with DeepSeek‑V3.2 (second place 35.15) and 46.39 with Qwen‑3‑30B (second place 41.94).

Retrieval Accuracy

On gold‑labelled tests, MemSifter’s NDCG@1 reached 70.00 on LoCoMo‑32K, far surpassing the next best ReasonRank (47.64), demonstrating the proxy’s precise filtering.

Ablation Studies

Removing the task‑result‑oriented RL component caused a performance drop of up to 26.80%, confirming its central role; omitting marginal‑utility or rank‑sensitive rewards also led to noticeable degradations.

Efficiency Analysis

Using a 4‑billion‑parameter proxy, MemSifter’s inference latency per query was 3982 ms, less than half of a 7‑billion‑parameter re‑ranking model. Compared to a 632‑billion‑parameter DeepSeek‑V3.2 handling a 128 K context directly, MemSifter’s latency was only 1/12, achieving orders‑of‑magnitude cost reduction.

4B proxy model, 3982 ms latency per query.

DeepSeek‑V3.2 (632B) latency ≈ 12 × higher.

5. Illustrative Cases

Three representative scenarios showcase MemSifter’s reasoning:

Long‑dialogue memory (LoCoMo) : The proxy correctly identified the conversation turn answering “When did John and his wife travel to Europe?” and ranked it highest.

User‑personalized memory (LongMemEval) : For the query “Where will I stay during my Hawaii birthday trip?”, the proxy retrieved the relevant planning turns.

Deep research (WebDancer) : Even with many semantically similar distractors, the proxy isolated the turn containing the precise answer to a complex knowledge question.

- GitHub open‑source address: https://github.com/plageon/MemSifter
- Paper URL: https://huggingface.co/papers/2603.03379