Why Mobile Trace Is Hard and How OpenTelemetry Solves It

Why Mobile Trace Is Hard and How OpenTelemetry Solves It

Mobile app development faces growing complexity as products evolve from small prototypes to large, stable releases, making problem reproduction and root‑cause identification increasingly difficult.

Why End‑Side Problems Are Hard to Reproduce

Inconsistent log collection between mobile and server sides.

Numerous modules, diverse frameworks, fragmented devices, and complex network environments hinder data gathering.

Lack of contextual correlation across different frameworks and systems.

Multiple business domains require cross‑team collaboration, raising manual operation costs.

Four‑step solution:

Establish a unified standard using a standard protocol to constrain data collection and processing.

Provide consistent data collection capabilities across platforms and frameworks.

Automatically associate and process data from multiple systems and modules.

Leverage machine‑learning for automated experience analysis.

Unified Data Collection Standard

We adopt OpenTelemetry (OTel), a CNCF‑driven observability standard that unifies trace protocols, supports multiple languages, and offers vendor‑agnostic collectors.

All mobile data collection follows OTel, with storage and analysis built on SLS LogHub.

Challenges of Mobile‑Side Data Collection

Data linking difficulty.

Performance assurance difficulty.

Preventing data loss.

Automatic Data Linking

OTel trace protocol requires consistent trace_id, parent_id, and span_id. On Android we use ThreadLocal for thread‑local context; for coroutines we propagate context via coroutine schedulers. On iOS we rely on activity tracing to bind context to activities.

Third‑Party Framework Data Collection

Two common approaches: instrument interceptors or use hooks when APIs are limited. Both have drawbacks such as incomplete coverage and invasive code changes.

We solve this by implementing a Gradle plugin that performs bytecode instrumentation (using ASM) to inject tracing code into third‑party libraries like OkHttp, ensuring low‑intrusion and automatic context linking.

Performance Guarantees

We implement core components in C, use Protocol Buffers for efficient serialization, apply configurable memory limits, dynamic memory management, and a ring‑file cache to reduce I/O. These optimizations double throughput and halve CPU/memory usage, supporting over 400 events per second.

Ensuring No Data Loss

We employ a write‑ahead log (WAL) to persist data locally before transmission, aggregate and compress batches with LZ4, retry on failure, and use edge‑node acceleration. This reduces packet size by 2.1×, improves QPS by 13×, and achieves 99.3% delivery success.

Multi‑System Data Correlation

Data from all endpoints is stored in SLS LogHub. Using OTel parent_id and span_id, we reconstruct trace trees, handling missing parents with virtual nodes, and aggregate by parent to produce ordered trace links.

Topological Generation and Batch Processing

We transform streaming into batch processing using MapReduce on SLS ScheduledSQL. The map phase groups by trace ID, then aggregates by span and parent IDs, producing edge information and dependency data for further analysis.

Automated Root‑Cause Localization

We extract features from trace data, cluster anomalies with HDBSCAN, and apply graph algorithms to pinpoint the start of abnormal traces, enabling automated root‑cause detection for any OTel‑compliant source.

Case Study: Multi‑Device End‑to‑End Tracing

A scenario links iOS (command sender), Android (command receiver), and backend services to trace a remote car‑air‑conditioner activation, showing end‑to‑end latency, request counts, error rates, and dependencies across devices.

Overall Architecture

Data sources: mobile SDKs implementing OTel.

Storage layer: SLS LogHub for unified data storage.

Processing layer: pre‑processing of key metrics, traces, dependencies, topology, and features.

Application layer: trace analysis, topology queries, metric queries, raw log access, and root‑cause localization.

Future Plans

Enhance plugin and annotation support to reduce code intrusion.

Enrich observable data sources with network quality and performance metrics.

Provide user‑access monitoring and performance analysis capabilities.

Open‑source core technologies to the community.