Skip to content
Building Automated Feedback Loops for Mobile App Performance, Memory & Reliability
Mobile Development 22 min read

Building Automated Feedback Loops for Mobile App Performance, Memory & Reliability

The article describes three automated feedback loops—performance, memory leak detection, and reliability—for a chat assistant Android app, using on-device tracing, heap analysis, and black-box testing to continuously measure, diagnose, and guard against regressions on real devices.

Thought Artisan
Thought Artisan
Thought Artisan

Why Loops Instead of One‑Off Tests

The chat assistant's critical path spans third‑party UI, accessibility framework, screenshot, OCR, model inference, and network calls—latencies that only appear on real devices. Vendor schedulers migrate tasks between big/little cores, kill background processes, and clear settings. Rather than a single measurement, the team built three feedback loops sharing a common skeleton:

Measure → Locate → Modify → Verify → Archive → Guard . The last step codifies checks so every code change re‑enters the loop at step one. Each cell in the diagram maps to a script, test, or file in the repository.

Performance Loop

2.1 First Measure: Where a Click Spends Its Time

Key paths were instrumented with Trace markers (zero allocation when not recording). A single click in WeChat "File Transfer Assistant" (cloud mode) breaks down as follows:

Cloud generation dominates: ~2.2 s total, 1.4–1.9 s waiting for the model—little the client can do.

Client‑controllable parts are milliseconds: chat capture (screenshot + OCR ~0.4 s), memory recall (bge embedding), accessibility callbacks.

Every stage has a performance budget: click‑to‑result P1, screenshot+OCR overhead P2, single accessibility event P4 (≤8 ms, must not drop frames), memory recall P5 (≤50 ms).

Numbers shown are per‑stage percentiles from a single run of three clicks—they cannot be summed. This variability is exactly why a loop is needed.

2.2 Turning Manual Tests into a Repeatable Closed Loop

Initially "click‑to‑result" was measured by hand ("roughly two‑three seconds"). To compare before/after changes, the team created perf-loop.sh:

Auto‑click: repeatedly taps the floating bubble in a designated chat, waits for result, presses Home to dismiss panel, repeats N times × M rounds.

Record traces: atrace captures each click's stage boundaries.

Summarize & compare to budgets: trace_summary.py computes P50/P95/max per stage; non‑zero exit if any budget is exceeded.

Archive: each round appends a line to perf-history.csv with the tested package's commit hash and install time. Two runs (before/after change) give a direct row‑by‑row comparison.

For CPU profiling, PROFILE=1 enables Perfetto 1 kHz call‑stack sampling of the process. Both atrace (stage boundaries) and Perfetto (stacks) use the BOOTTIME clock ; perfetto_analyze.py assigns each sample to its stage. A --html flag later added per‑stage interactive flame graphs in a single standalone file.

2.3 Letting Call Stacks Explain Why the Memory Block Is Slow

In a microbenchmark, bge embedding for a 12‑line query takes 52 ms ; on‑device the memory block takes ~140 ms (2.7× slower). Call‑stack sampling revealed:

Almost all time is in embed.forward (bge inference itself).

34 % of samples land in MNN thread‑pool sched_yield —threads spinning while waiting for work.

During the same window the vendor scheduler pauses big cores .

The issue: four threads busy‑waiting combined with the scheduler withholding big cores. Microbenchmarks run on idle phones and miss this. Thread count and power hint were hard‑coded in the native library; they were exposed via JNI and libembedjni.so rebuilt. Back‑to‑back A/B on the same device (same chat, 6–8 clicks per config, run consecutively to keep temperature/background state stable) yielded:

More threads ≠ better: 6 threads had worst P95 (243 ms); 2 threads slower than 4.

Final choice: 4 threads + Power_High , P95 dropped from ~190 ms to ~137 ms.

A microbenchmark alone would likely have suggested adding threads—the opposite of the real‑device optimum.

2.4 Optimizations Considered but Not Done (With Reasoning)

The loop also documents deliberate non‑actions:

Pushing memory block to 50 ms budget: config already maxed; further reduction would require shortening the query window (now last 12 lines) or pre‑computing, hurting recall—a product trade‑off, not engineering.

Loading animation during generation: consumes ~17 % of a core in cloud mode; in local mode it competes with on‑device inference. Decision deferred.

OCR pre‑crop of screenshot: estimated 14 % pixel reduction (~50 ms saved) but required coordinate translation for sender detection; edge messages could be clipped. Risk outweighed benefit.

The last item was later revisited. After the screen‑reading refactor, message bubble boundaries became available at capture time. The crop was implemented in captureScreenshot (keyboard removed, union of all bubbles) via PR #75. Motivation shifted to accuracy : status bar, input bar, and surrounding panels are removed before OCR, preventing mis‑recognition. The earlier coordinate‑translation risk was covered by seven unit tests. Same problem, opposite conclusions—because the preconditions changed.

Memory Leak Loop

3.1 Approach: Stress Lifecycles, Compare Heap Growth Trends

The app repeatedly creates/destroys: settings Activity/Compose UI, floating bubble/panel windows, per‑click screen‑reading and generation flows. Leaks manifest as old instances retained by a field, accumulating each cycle. A single heap dump cannot tell if 300 objects are excessive. The leak loop's core is trend comparison : leak-loop.sh per round: open home → press Home; disable accessibility until bubble disappears, re‑enable until it returns; click bubble once in test chat. Warm‑up 2 rounds (first‑use caches excluded), then every 10 rounds dump Java heap via Perfetto's android.java_hprof. heap_leak.py applies four rules: only GC‑reachable objects; a class must increase in every interval and average ≥1 new instance per round to be flagged; singleton lifecycle objects ( BubbleController, FloatingBubble, MainActivity) must have ≤1 instance at any time—if more, print shortest reference chain from GC root (field names de‑obfuscated via R8 mapping); finally group by the last few hops of the chain to see who holds the growth.

3.2 First Run: Growth Comes from Framework, Not App Code

After warm‑up + 2×10 rounds, reachable objects grew from 149,441 to 151,190 (~105 KB). Every increase was accounted for:

OkHttp HTTP/2 HPACK dynamic table (capped at 4 KB default)—not a leak.

Window Binder stubs ( ViewRootImpl$W etc.): GC root is JNI global ref; ViewRootImpl itself stable at 2 instances. Cleaner (pending native allocations) and Long (ART internal).

The scariest line: ChatAccessibilityService instances 3 → 7 → 9 . Reference chain shows holder is framework's IAccessibilityServiceClientWrapper.mContext —a system_server Binder stub. Each accessibility restart creates a new service instance; old ones wait for system to release the Binder. Growth is not one‑per‑round, indicating gradual release. Zero references from app code.

First‑run conclusion: no app‑side leaks.

3.3 Peak Memory Optimizations

Beyond leaks, peak memory matters. Full‑screen screenshot → software bitmap ≈13 MB. Three changes:

Move screenshot callback to background thread (copy no longer blocks main thread).

When user clicks rapidly and previous request is cancelled, recycle the copied bitmap immediately instead of waiting for GC.

After cropping the screenshot, recycle the full‑screen bitmap at once , so OCR phase holds only the cropped image.

Reliability Loop

4.1 Four Sources Cross‑Verifying Each Other

White‑box: guards logic (DB transactions, migrations, timeouts); runs every CI, seconds.

Black‑box: guards recovery (crashes, memory pressure, permission revocation, network loss, rotation); runs on‑demand on real devices.

Online diagnostics: guards user devices—error codes, crash logs, exit reasons, key‑operation latencies.

Vendor systems: behaviors tests cannot cover, only encountered on real devices.

4.2 Making Black‑Box Tests Validate Diagnostics

Diagnostics are valuable only if they surface problems. Black‑box tests incidentally verify diagnostics :

Crash test uses am crash to kill the process; exports diagnostics before/after.

Requires APP-901 (crash synchronously written in uncaught exception handler) to appear exactly once more .

Requires new APP-902 with latest exit reason CRASH (read from ApplicationExitInfo on next start).

Every black‑box case ends by querying dumpsys dropbox for crashes/ANRs in that window. Crash case expects exactly one—this also proves the detection itself works. The test device blocks logcat for release builds; relying on logs would make all cases "pass". Diagnostics also include latency distributions : click‑to‑result, cloud/local generation, screenshot+OCR, one learning cycle—each bucketed into 9 fixed ranges (<250 ms … ≥30 s) with counts, exporting P50/P95. Instrumentation only on click path and learning, not on accessibility callbacks, to avoid hot‑path overhead. When users report "slower", the exported distribution is ready for analysis.

4.3 Reliability Recovery Baselines

Measured 11‑15, two consecutive 6/6 passes:

Accessibility service restarts in 65 ms.

Floating window service restarts in 80 ms.

Accessibility re‑bind after toggle: 52 ms.

Network test ("test connection") fails with "network connection failed" in 1.6 s—no infinite spinner.

4.4 What Tests Miss: Vendor "Swipe‑Away" Behavior

All black‑box cases passed, yet a real‑device issue surfaced: swiping the app from recent tasks on Honor kills the process and clears the accessibility setting . User reopens chat—floating bubble gone, must manually re‑enable accessibility.

This is hard to test: depends on vendor policy, varies by brand/version. Two mitigations:

Foreground service keeps bubble alive: KeepLiveService with persistent notification, tied to bubble lifecycle. Not sticky; after kill, it restarts when accessibility reconnects and bubble returns. Start failure logs BUB-501.

Guide user to "allow background run" : no public API for vendor auto‑start managers. Home screen adds an entry that sequentially tries Honor, Huawei, Xiaomi, OPPO/OnePlus, Vivo settings pages; falls back to app details. State unreadable, so shown as button + explanation, no checkbox.

The loop closes here too: the fix records BUB-501 in diagnostics so future occurrences are visible.

Running Loops on a Daily‑Driver Phone

All three loops share a prerequisite: the test device is a real person's daily phone —real chats, real API keys, real system settings. Automation mistakes have real consequences. Safeguards evolved:

Clicking wrong chat: once a manual test clicked a real private chat (input method hidden, fixed coordinate tapped). Now perf-loop.sh verifies chat title before each click: same chat → 0 % title change, different chat → ~7 %; threshold 0.2 %. Mismatch aborts the whole round. Only title bar screenshot kept; full screen never saved.

Accidental candidate selection when closing panel: coordinate‑based close button moves when title wraps to two lines, causing a candidate tap that auto‑fills the input. Solution: always press Home to dismiss—leaving the chat app hides bubble and removes panel safely.

Changing system settings: leak loop disables accessibility; script records original value at start and restores on exit (including mid‑run failure).

Monkey and force-stop : random taps hit "clear memory"; on this ROM force-stop removes the app from enabled accessibility services. No monkey; crash scenarios use am crash.

Test framework interfering with subject: UiAutomator connecting to UiAutomation by default suspends other accessibility services, including the app under test. Must set FLAG_DONT_SUPPRESS_ACCESSIBILITY_SERVICES before first use.

Closing Thoughts

Agents are already highly capable; guided well they produce excellent results—code agents are power tools, and combined with a software engineer's fundamental knowledge, they become unstoppable.

Building a memory leak test loop, using Perfetto to collect Java heap, then analyzing for intrinsic leaks.

Appendix: Data & Scripts

Repository: https://github.com/zhaojunmin/aichat/

Performance loop & call‑stack sampling: quality/scripts/perf-loop.sh, perf_guard.py, trace_summary.py, perfetto_analyze.py, quality/perfetto/tap-profile.pbtxt

Memory leak loop: quality/scripts/leak-loop.sh, heap_leak.py, quality/perfetto/java-heap.pbtxt

Reliability black‑box: quality/reliability/, quality/scripts/run-reliability.sh

Methods & baselines: docs/performance-testing.md, docs/reliability-testing.md, docs/observability-design.md

Article figures: docs/article/quality-loops/build_figures.py (SVG → 2× PNG via cairosvg)

Test device: Honor ELP‑AN00 (Snapdragon 8s Gen3 / Android 16)

Original Source

Signed-in readers can open the original source through BestHub's protected redirect.

Sign in to view source
Republication Notice

This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactadmin@besthub.devand we will review it promptly.

Androidautomated testingperformance testingreliability testingmemory leak detectionPerfettofeedback loopsatrace
Reader feedback

How this landed with the community

Sign in to like

Rate this article

Was this worth your time?

Sign in to rate
Discussion

0 Comments

Thoughtful readers leave field notes, pushback, and hard-won operational detail here.