Building a Privacy-First Chat Assistant with On-Device LLMs on Android
A developer shares the technical journey of building a floating chat assistant for Android that suggests replies using on-device LLMs, detailing accessibility-based screen reading, OCR fallback for WeChat, a local memory system, CPU vs GPU benchmarking surprises, and AI-assisted development with Claude Code.
Product Overview
The "Chat Assistant" is a floating overlay ball for apps like WeChat, Soul, and Momo. When a user taps it during a conversation, the AI generates 1–5 short, style-varied reply suggestions that can be inserted directly into the input field. Target users include socially anxious people, those who struggle with wording, and anyone concerned about chat privacy. The assistant works in three steps: grant accessibility and overlay permissions, open a chat screen and tap the floating ball, then pick a suggested reply.
Key Features
Reads chats – Uses Android AccessibilityService plus on-device OCR (ML Kit) to extract conversation text and distinguish sender vs. receiver.
Generates relevant replies – Built-in personas (dating, ice-breaking, business, support) plus custom tone; replies are forced to be short and colloquial.
Learns the user over time – A global memory system permanently stores extracted profile/facts/preferences/style (append-only, latest-wins conflict resolution). Only the most recent N memories are injected into the prompt to limit context length. A management UI lets users edit or delete individual memory entries.
Works offline – Privacy mode runs a small Qwen2.5 model locally via MNN; zero data leaves the device.
Development Process: From Screen Reading to Local LLMs
1. Extracting Chat Content from Other Apps
The foundation is Android's AccessibilityService, which reads the foreground window's view hierarchy. Reality varies by app:
Some apps expose clean text in the view tree, allowing direct parsing and geometric separation of "me" vs. "them".
WeChat and others harden/obfuscate the view tree. The fallback is screenshot + on-device OCR (ML Kit Chinese recognition, bundled model).
WeChat-specific OCR tricks:
Use the characteristic green bubble background color to identify the current user's messages (more reliable than left/right position).
Discard everything above the first date/time stamp (profile cards, etc.).
Critical bug fix: When the soft keyboard appears, its keys get OCR'd as chat text. Solution: obtain the input-method window bounds via accessibility (TYPE_INPUT_METHOD) and crop that region out before OCR.
2. "Gets to Know You": Global Memory System
Every user utterance is appended to a local text file. Once enough lines accumulate, a large model summarizes them into structured "profile/facts/preferences/style" stored in a local database. These memories are injected into the prompt at generation time.
Two deliberate design decisions:
Append-only, never rewrite – Each learning round only adds new entries or overwrites conflicts with the latest version. Full rewrites are avoided to prevent a model hallucination from erasing prior facts.
Inject only recent N memories – The full history is retained on device, but only the most recently updated entries are fed to the model, preventing unbounded prompt growth.
A memory management UI allows per-entry editing and deletion to correct model mistakes.
3. Slimming Down & "De-geeking" the UI
Early versions exposed a long list of technical settings (Base URL, package names, node trees). For real users, the UI was split into two screens: a home screen with daily-use items (permissions, floating ball, AI service, reply style, memory) and an "Advanced Settings" screen for technical/debug options. Build optimizations: R8 code shrinking, resource shrinking, keeping only ARM native libraries. APK size dropped from 59 MB to 21 MB.
4. Putting an LLM on the Phone
Cloud LLMs (DeepSeek, Doubao) work well but send chat content — including the other party's messages — to third-party servers, which is a privacy/compliance non-starter. The solution: on-device inference.
Framework: Alibaba's MNN (well-optimized for Qualcomm/ARM, native Qwen support).
Model: Qwen2.5 (strong Chinese, available in 0.5B/1.5B/3B sizes).
GPU vs. CPU Benchmark (Snapdragon 8s Gen3, Adreno 735)
Conventional wisdom says GPU should be faster. On-device tests with the 1.5B model:
CPU (4 threads): ~28 tok/s, correct output.
GPU OpenCL (fp16): Slightly faster, but produces garbage (numeric overflow).
GPU OpenCL (fp32): Slower than CPU, correct output.
Counter-intuitive conclusion: On this chip, GPU provides no benefit for small models. The fast fp16 path suffers numeric overflow producing garbage; the correct fp32 path is slower than CPU. Modern Snapdragon CPUs have i8mm/dotprod instructions, and MNN's CPU integer kernels already saturate the hardware. GPU support was dropped, saving 80 MB of OpenCL libraries.
China-Specific Distribution Hurdle
Models were hosted on Hugging Face. The dev machine (outside China) downloaded quickly; real devices in China could not resolve huggingface.co (dead IP). Switching to ModelScope (魔播社区) enabled a 0.5B model (~540 MB) to download in 40 seconds on a real device.
Multi-Size Model Selection
Three options shipped with real-time download progress (percentage, current file, downloaded size):
0.5B – fastest
1.5B – balanced (~28 tok/s)
3B – best quality (~15 tok/s, half the speed of 1.5B)
Users choose based on device capability and preference.
Epilogue: AI-Assisted Development
The entire pipeline — screen parsing, memory system, local model integration, on-device benchmarking, and bug hunting — was built in collaboration with Claude Code . The AI assistant could read/write code, compile, drive adb for real-device screenshot validation, and even run the LLM benchmarks that yielded the "GPU useless" finding. One developer + one AI partner took "screen reading + floating window + cloud/local dual inference + memory" from zero to a pocket-sized app. Tools will get smarter, but applying them to real scenarios remains the most interesting part.
Signed-in readers can open the original source through BestHub's protected redirect.
This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactand we will review it promptly.
How this landed with the community
Was this worth your time?
0 Comments
Thoughtful readers leave field notes, pushback, and hard-won operational detail here.
