From Zero Tests to 165: Android Chat Assistant Quality Engineering Retrospective
The team transformed an Android chat assistant with zero tests and CI into a robust product through 21 PRs, establishing a five-layer safety net, enforcing privacy-first design, optimizing performance via trace-driven analysis, hardening reliability with black-box testing, and adding on-device observability—all while uncovering flaws in their own test infrastructure.
Background: A Sensitive App
The "Chat Assistant" feature reads private chat content, holds high-risk permissions (accessibility, overlay, screenshot), parasitizes on other apps (WeChat, QQ) whose UI changes break screen-reading rules, must respond within 3 seconds while the user waits, and runs as a resident service where an accessibility crash disables the service until the user manually re-enables it. Yet the engineering state had 0 tests, 0 CI, 0 static checks .
Prioritizing Quality Attributes
Quality attributes often conflict (e.g., testability abstractions may slow hot paths; observability logs may leak privacy). The team first wrote down a priority order to arbitrate conflicts:
Privacy first — even at the cost of some functionality and convenience, chat content must never leave the device unintentionally.
Every attribute expressed as verifiable scenarios — not "fast" but "click-to-result P50 ≤ 2.5s, P95 ≤ 5s"; not "secure" but "in local mode, 0 bytes of chat content sent to network".
Observability last but not zero — later this "catch-up" revealed many issues.
Build the Safety Net First, Then Refactor
Refactoring without tests is surgery blindfolded. The first PR only set up unit tests, Lint, and CI. They evolved a five-layer safety net:
JVM unit tests + Robolectric — run on every commit in CI; logic correctness; grew from 0 to 165 tests .
Microbenchmarks / macrobenchmarks — on real devices; measure code speed, memory allocation, startup and scroll smoothness.
End-to-end trace — on real device release builds; latency of each segment: accessibility event, click-to-result.
Reliability black-box — on real device release builds; crash, memory pressure, permission revocation, network loss, config changes — can it recover?
On-device diagnostics — on user phones; how we learn when something goes wrong.
Guardrail agreements (more important than tools):
Local test + lint + assemble must all pass before opening a PR.
Real-device verification uses release builds — R8 obfuscation and shrinking expose keep-rule issues debug builds miss.
PRs touching the screen-reading hot path must attach trace data proving no regression.
Lint uses a baseline — accept existing issues, only block new ones, avoiding hundreds of warnings at once.
Each PR does one thing, independently revertible; all branches based on main, no stacking.
Security & Privacy: Every Boundary Guarded by Code and Tests
4.1 Local Mode: 0 Bytes Leave Device
User selects "local mode" meaning chat content must not leave the phone. A subtle path existed: when the local model isn't ready, should we temporarily fall back to cloud? Answer: absolutely not . Local model not ready → error, prompt user to download model. This invariant is locked by unit tests: even with a cloud key configured, local mode never calls cloud.
4.2 One Review Uncovered Four Privacy Issues
First code review against privacy goals found: allowBackup="true" — memory database and API key would follow system cloud backup and device migration. Fix: disable backup, use dataExtractionRules to exclude database, prefs, files.
API key stored plaintext in SharedPreferences — root or data export leaks it. Fix: Android Keystore AES-GCM encryption; legacy plaintext key auto-migrates on first read.
Debug "dump view hierarchy" running in production — chat plaintext resident in memory, occupying main thread. Fix: run only when debug switch enabled.
Local-mode memory learning never runs — condition checked cloud key ; pure local users' memory feature silently broken. Fix: "local mode checks model readiness, cloud mode checks key".
The last isn't a privacy leak but typical: local-mode users are the most privacy-conscious, yet their feature was broken with no one knowing.
Key encryption did not use EncryptedSharedPreferences (deprecated); instead direct Keystore-generated key with AES-GCM, <100 lines, fully unit-tested including "tampered ciphertext refuses decryption" and "undecryptable clears key without crashing app".
4.3 Observability Privacy: Enforced by Interface, Not Discipline
Adding diagnostic logs later reintroduces privacy risks: logs accidentally capture chat content, keys, server error bodies. Solution: interface design prevents it :
Logging interface accepts only error codes and numeric fields — no string parameters.
Exceptions retain only class name , not message (server errors may echo request content).
A "sentinel" test : embed unique strings in key, base URL, persona, exception message; run full record-persist-export flow; assert zero sentinels appear in export.
Performance: First See Which Thread Code Runs On
5.1 One Thread Map Revealed Four Hotspots
Before optimizing, they drew a runtime view: one click on the floating ball, what each thread does.
Most counter-intuitive was ④ :
Accessibility service reads another app's view hierarchy; every node access is a cross-process call answered by the chat app's own main thread .
Original implementation: "on every accessibility event, parse the entire view tree" — only used when user clicks floating ball, rest discarded. User scrolling chat list generates dozens of events per second. Moving parsing to background thread only saves our main thread; the other app's main thread is still blocked by us .
Real fix: change when we parse, not which thread — parse once on floating-ball click ( #23). Three side benefits:
Reads screen at click moment, no stale content from previous event.
Process killed then immediate click won't show "no content read".
Chat plaintext no longer resident in memory.
5.2 Trace-Driven Optimization
After change, added Trace markers on critical path, captured with atrace on release build. Data showed each accessibility event still took 7–8ms (P95 17–20ms), ~ 95% of time spent in rootInActiveWindow cross-process call .
Further analysis: scroll events from same foreground app cannot change "who is foreground", so skip them. Result: main-thread events during scroll dropped from 30–42 to 0 ( #24).
Figure 6 part A: click-to-result ~2.7s, model generation 2.4s , client work only ~8%. Client optimization goal not "reduce total latency" but " don't block main thread ": floating window, accessibility callbacks, settings UI all share one main thread.
5.3 A Near-Merge "Optimization": Sacrificing Hot Path for Testability
During parser testing, one approach: copy entire view tree to plain data objects, then feed parser — enabling JVM tests. PR written but rejected : it forced full-field access and object allocation on every node, whereas original only accessed fields on rule-matched nodes.
Rule codified: testability refactors must not add runtime overhead to hot paths .
5.4 Benchmarks Can Lie
First microbenchmark: "bge model query vector 93ms" over budget. Root cause: benchmark used 30 chat lines as query, actual code uses 12. Corrected: 52ms . Measurement itself must be validated.
Reliability: Failure Is Normal, Key Is What Happens After
6.1 Timeouts for Both Cloud and Local
Cloud : errors categorized (invalid key, insufficient balance, rate limit, wrong address, server error, timeout, network error) with actionable Chinese messages; requests cancellable; overall 30s deadline.
Local : native inference is a blocking call uninterruptible from Kotlin. Timeout moved inside engine: in decode loop, check elapsed per token, stop at deadline, reset engine before next call.
6.2 Testing Revealed: Empty Reply Rendered as Blank Candidate
Table-driven tests for "model returns malformed content" caught two real bugs:
Model returns empty string → panel shows blank candidate .
Model returns [] → panel shows candidate with literal content [].
Now both explicitly show "response content unavailable".
6.3 Learning: Turn Two Steps into One
One learning round does two things: write learned result to memory DB, advance "learned up to which reply" cursor. Originally separate: former in Room, latter in SharedPreferences with async apply().
Process killed between steps → same batch re-learned → duplicate entries in memory.
Fix: move cursor into same DB, write both in one transaction. Tests with Robolectric on real Room DB cover "write half fails" and "write succeeds but cursor save fails" — both assert DB rolls back.
DB version bumped v4→v5, added migration tests: create DB with v1–v4 schema, open with current version. To verify migration tests actually work, did mutation testing : deliberately drop primary key on v5 new table — all 4 migration cases failed.
A test that never fails is as good as no test.
6.4 Real-Device Black-Box: Six Destructive Scenarios
quality/reliabilityis a separate test module driving release build externally via shell commands:
Process crash ( am crash ) — both services auto-restarted by system, accessibility switch not turned off. Measured: accessibility 31ms–0.6s, floating window 18–80ms recovery.
Memory pressure (foreground + background three levels each) — no crash, no process restart. ✅
Revoke overlay permission while running — no crash; home screen prompts "enable"; restored after re-grant. Recovery 21–63ms.
Disable accessibility service while running — no crash; re-enable triggers system rebind. 24–64ms.
Offline then tap "test connection" — cloud reports "network connection failed" within 10s, no infinite spinner. ~1.6s.
Dark mode + landscape — no crash, no restart. ✅
Each case ends by checking dumpsys dropbox for crash/ANR records. Crash case expects exactly 1 record, proving detection works.
Deliberately omitted : random monkey stress — it would tap "clear memory" buttons destroying real data.
6.5 Black-Box Tests' Own Pitfalls
Most interesting: first test run's "good results" were fake.
"Crash recovery 63ms" was fake data. UiAutomator connecting to UiAutomation by default pauses all other accessibility services on device , including the one under test. First detection read dumpsys activity services which retains service record after unbind, masking the issue. Fix: set FLAG_DONT_SUPPRESS_ACCESSIBILITY_SERVICES, use dumpsys accessibility bound services.
Test device blocks release build logcat. No app logs visible; crash detection switched to system dropbox; debug coordinates via Trace.beginSection.
Reading dropbox occasionally returns empty string. If treated as "no crash", test silently passes real crash. Fix: require fixed header in output, else retry.
Click immediately after scroll misses. List still in fling; click only stops scroll. Must wait for settle, then re-locate element.
Correctness: Issues Only Visible on Real Devices
Running full flow on real device revealed two recognition errors unit tests could never catch:
Issue 1: One bubble split into multiple messages. WeChat screenshot OCR (ML Kit) returns each wrapped line of a bubble as independent text block. Fix: merge adjacent lines meeting both position and color conditions. Debugging uncovered two traps:
OCR text box left edge jitters ~20px per run; half-line-height alignment tolerance caused missed merges.
Original "merge then filter" caused timestamps to stick into adjacent blocks, breaking timestamp filter.
Issue 2: Group chat speaker name mistaken as peer message. Identify name by position, font size, background color. Another trap: a light wallpaper #E7F0F9 had color difference 45 from white bubble, while merge tolerance for anti-aliasing was 48 → judged "same background". Comparing solid backgrounds should use stricter tolerance.
Observability: How We Know When User Has a Problem
Design doc before implementation, review decided four things:
Temporarily no stack traces (release obfuscation needs mapping file).
Package name only in screen-read and OCR records .
Two 256KB rotating files .
Default-on switch, off deletes data .
Implemented in four PRs:
Core : 34 error codes ( domain-three-digit, immutable once published; tests against committed snapshot prevent changes); unified logging entry; crash writes sync record then delegates to system handler; startup reads ApplicationExitInfo for previous process exit reason — captures native crashes and ANRs that in-process handlers miss.
Integration : 28 sites changed from silently swallowing exceptions to "log error code then degrade".
Export : preview shows full text; content leaves phone only on user "share" or "copy".
Latency distribution : on-device P50/P95 for key ops like click-to-result.
Integration uncovered most valuable find : floating window service startForeground failure silently swallowed by runCatching. Android 12+ rejects background-started foreground services — a prime cause of "floating ball mysteriously disappears", previously zero trace .
Observability folded into reliability tests: crash case now reads diagnostic export before/after crash, asserts the sync-written crash record and the restart's "last exit reason = crash" both exist.
Issue Catalog
All discovered issues grouped by detection method:
Code review against quality goals : local-mode memory learning blocked by cloud key (reliability); allowBackup, plaintext API key, debug dump in prod (privacy); accessibility callback no exception guard → service disabled (reliability).
Writing unit tests : empty reply and [] rendered as candidates (reliability); learning cursor and memory write not in same transaction (reliability).
Runtime view analysis : per-event full tree parse occupying chat app main thread (performance); memory vector compute on main thread; OCR bitmap copy on main thread (performance); local mode learning and generation contending same inference lock (performance).
Trace data : 95% event time in one cross-process call (performance).
Macrobenchmark : profileinstaller 1.3.1 doesn't support Android 14+, startup profile not installed (performance).
Real-device end-to-end : screenshot splits one bubble into many; group name mistaken as message (correctness).
Writing diagnostic integration : startForeground failure silently swallowed (observability).
Test's own issues : benchmark query length wrong (93ms→52ms); UiAutomator paused tested accessibility service → fake 63ms recovery; automation script fixed-coordinate close panel, panel height change caused mis-click on first candidate (draft only, not sent) — led to rule: real-device tests close panel by pressing Home, never tap screen.
Takeaway Lessons
Write priorities first, code later. Quality attributes conflict; a written priority table ends debates fast.
Every goal must be quantifiable. "Fast" and "secure" can't be verified; "P95 ≤ 5s" and "0 bytes off device" can.
Measure first, optimize later. Trace told us 95% time in one cross-process call; without it we'd optimize irrelevant code.
Hot-path constraints trump testability. Slowing every user scroll for easier testing is a bad trade.
Verify on real devices with release builds. OCR issues, ROM log blocking, UiAutomator pausing accessibility — only surface on real hardware.
Privacy by interface design, not discipline. Logging interface rejecting text is far more reliable than code-review reminders "don't log".
Tests lie too. Mutation testing, "exactly 1 crash" positive controls, "empty result retry" — all are tests for tests.
Numbers at a Glance
21 PRs ( #14 – #34)
Unit tests 0 → 165, CI all green
Real-device reliability black-box 6 scenarios, consecutive multi-run 6/6
Scroll-time main-thread accessibility events 30–42 → 0
34 error codes, 28 integrations, 27 previously silent exceptions now recorded
Local mode chat content: 0 bytes off device
All chat examples in this article are fictional.
Signed-in readers can open the original source through BestHub's protected redirect.
This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactand we will review it promptly.
How this landed with the community
Was this worth your time?
0 Comments
Thoughtful readers leave field notes, pushback, and hard-won operational detail here.
