From Recency to Semantic Retrieval: Building On-Device Memory for Android AI Assistant
The article details the evolution of a memory system for an Android screen-reading AI assistant, from naive recency-based injection to on-device semantic retrieval using embeddings, covering research, lexical vs embedding benchmarks, model selection (bge-small-zh), runtime comparison (MNN vs LiteRT), quantization choices, and a fallback strategy ensuring zero regression.
Why Memory for an On-Device Chat Assistant
The assistant reads the current chat screen via Android accessibility and generates replies in the user's voice. Without long-term memory it cannot distinguish between a hiking enthusiast and a homebody — both would receive the same reply to "Free this weekend?". The solution requires a long-term memory layer that stores persona, facts, preferences, and speaking style, with hard constraints: on-device, offline, privacy-preserving, low latency, and minimal token cost.
Naïve Starting Point: Recency-Based Injection
The first version simply logged every user reply, periodically distilled batches into memories via an LLM, and injected the most recent N memories at generation time. This worked but suffered from a fundamental flaw: recency ≠ relevance . When discussing food, the system might inject last week's work-related memories. As the memory pool grew, dialogue quality degraded sharply.
Research-Backed Skeleton: Layered Memory + RAG Retrieval
A deep-search literature review yielded four pillars:
Short-term / long-term layering : long-term memory follows four stages — extract → update → retrieve → inject.
RAG-style on-demand retrieval instead of full injection — the key lever for saving tokens and reducing latency.
Three-factor scoring for retrieval: recency + importance + relevance (each normalized within the candidate set and summed), not just vector similarity.
Conflict-aware updates with ADD / UPDATE / DELETE operations (inspired by Mem0).
First Implementation: Three-Factor Scoring + Soft Delete
Replaced "most recent N" with three-factor scoring. For updates, adopted Mem0-style ADD/UPDATE/DELETE, but because LLM-judged deletion is risky (especially with small on-device models), DELETE was implemented as soft delete — items move to a trash bin, recoverable, with human-in-the-loop fallback.
The critical open question: how to compute relevance ?
Key Decision 1: Start with Local Lexical, Abstract the Scorer Interface
The simplest relevance is local lexical matching : Chinese bigram tokenization, compute token overlap. Zero dependencies, fully on-device, zero extra latency. Why not jump straight to embeddings? The team chose to get the three-factor framework running and validated first , making the scorer a pluggable interface so embeddings could be swapped in later.
interface RelevanceScorer {
fun scores(items, query): DoubleArray // query encoded once
}
├─ LexicalRelevance (token overlap)
└─ EmbeddingRelevance (vector cosine)This "start simple, leave an interface" decision saved massive rework later — changing models or runtimes never touched the scoring framework.
Data-Driven Evaluation: Is Lexical Enough?
Intuition said lexical might suffice for short Chinese queries, but intuition doesn't count. The team built a test set mirroring real-device style: 40 memories + 28 queries, automatically split by whether query and answer share surface tokens:
Lexically reachable : 13 queries
Purely semantic : 15 queries (nearly half)
Results (hit@1):
Method | Overall | Lexical-reachable | Pure-semantic
Lexical (bigram)| 0.429 | 0.885 | 0.133
bge-small-zh | 0.893 | 0.885 | 0.933Conclusion : Lexical matches lexical-reachable queries well (0.885) but fails almost completely on pure-semantic ones (0.133). Real dialogue is ~50% pure-semantic — e.g., "Southerner or northerner?" must retrieve "hometown Chengdu"; "All-night gaming" must retrieve "dislikes staying up late". Overall hit@1 jumps from 0.43 → 0.89 . Embedding is justified, evidence is conclusive.
Model Selection: A Candidate That Flipped
Several on-device candidates were evaluated on the same corpus (quality independent of runtime, tested on PC):
Model | Params | hit@1 | Pure-semantic | Notes
bge-small-zh | 24M | 0.893 | 0.933 | Small & accurate, winner
gte-small-zh | 30M | 0.857 | 0.933 | Runner-up
Qwen3-Emb-0.6B | 0.6B | 0.893 | 0.967 | Ceiling but too heavy
EmbeddingGemma | 300M | 0.857 | 0.867 | Large
all-MiniLM-L6-v2 | 22M | 0.214 | 0.133 | ❌ English model, fails Chineseall-MiniLM-L6-v2is a common tutorial default, yet underperforms lexical overall (hit@1 0.21) — English-trained models simply don't work for Chinese. bge-small-zh wins with the smallest footprint and near-top quality.
Key Decision 2: Runtime Intuition Overturned by On-Device A/B
Initial bias favored LiteRT (Google's on-device stack, future NPU support, growing ecosystem). Before committing, a quick on-device A/B compared MNN vs LiteRT on the same device (Snapdragon 8s Gen3) with the same model (Qwen2.5-1.5B):
Data flipped the intuition: on this device (not on LiteRT's official NPU whitelist), LiteRT had no NPU acceleration, its official 1.5B build only offered q8 quantization (weight bandwidth 2× q4, and decode is bandwidth-bound), and GPU only speeds up prefill — useless for short replies. MNN was 1.7–2.5× faster. Decision corrected from "prefer LiteRT" to "unify on MNN": LLM stays on MNN, embedding also uses MNN — one runtime, zero extra core libraries.
Deployment: bge on MNN
Converting bge from ONNX to MNN hit two snags and fixed a quantization policy:
Pitfall 1 : Off-the-shelf ONNX used fused MultiHeadAttention operator — unsupported by MNN. Solution: re-export from Hugging Face.
Pitfall 2 : Export must use eager attention; otherwise NaN / fused operators cause conversion failure.
Quantization : int8 vs fp16 (cosine similarity vs PC baseline):
Variant | Size | Cosine vs PC
fp16 (chosen) | 47.5MB | 1.0000
int8 | 57MB | 0.83 ❌int8 is too sensitive on this small model's attention weights — cosine drops to 0.83 and size is larger (embedding table not effectively quantized) → fp16 adopted .
Engineering reuse: encoder uses MNN core APIs, CLS pooling + L2 normalization, tokenization in Kotlin. Only one extra ~260KB native library.
On-device verification (Snapdragon 8s Gen3): model loads, 10/10 memory vectors encoded, semantic retrieval works, stable runtime. When unavailable (non-arm64 / model missing) it falls back to lexical with zero regression .
Final Architecture Overview
Memory retrieval evolved from "stuff by time" to "on-device semantic recall", fully offline and private. Added cost: ~260KB native lib + one fp16 model.
Six Evolutionary Lessons
Start simple + leave interfaces > big bang. Lexical first got the framework running; scorer pluggability meant model/runtime swaps never touched scoring logic.
Let data drive, not intuition. Three intuitions ("lexical enough", "all-MiniLM works", "LiteRT better") — two disproven by data. Key: split evaluation along dimensions that expose differences (lexical vs pure-semantic), otherwise averages deceive.
Calibrate priors with real measurements. "Prefer LiteRT" was a reasonable prior, but on-device A/B showed it slower on this NPU-less device.
On-device quantization must be measured. int8 looked attractive but cosine fell to 0.83 with larger size; fp16 was correct.
Fallback-first, zero-regression rollout. Entire embedding chain strictly falls back to lexical until model is ready; merging half-finished code into mainline doesn't change production behavior.
Reuse beats introduce. Device already had MNN; let LLM and embedding share it rather than pulling in another runtime — only 260KB extra.
All numbers come from actual recall benchmarks and on-device A/B tests. The project is an Android on-device AI reply assistant (screen-read + floating window + cloud/local dual inference + personal memory).
Signed-in readers can open the original source through BestHub's protected redirect.
This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactand we will review it promptly.
How this landed with the community
Was this worth your time?
0 Comments
Thoughtful readers leave field notes, pushback, and hard-won operational detail here.
