How to Build Smart, Scalable Data Tracking Solutions for Comprehensive Analytics

Data collection is the foundation of data analysis, and tracking (埋点) is the primary method; this article examines the challenges of data tracking and shares practical approaches for building effective solutions.

1. Common Tracking Implementation Schemes

1.1 Basic Process

The typical workflow from business requirement to production deployment includes requirement gathering, design, development, testing, and launch.

1.2 Comparison of Common Tracking Schemes

Code Tracking : Define business‑important events and manually add tracking code. Suitable for behavior analysis driven by business value. Advantages: selective collection, richer business data. Disadvantages: higher development effort.

Full Tracking : Pre‑integrate a set of generic events. Suitable for collecting baseline data from interfaces. Advantages: simple and quick, less development work. Disadvantages: limited data dimensions, cannot meet personalized needs.

Visual Tracking : Pre‑integrate events and select needed ones via a visual UI. Suitable for UI‑level user actions. Advantages: low development effort during operation. Disadvantages: tightly coupled with system framework, lacks business‑level interpretation.

2. Pain Points and Requirements

Balancing comprehensive data support with resource consumption is a core dilemma; an ideal smart tracking solution should be "anytime, anywhere, fully controllable".

Anytime, anywhere: support any business scenario and provide required data.

Cloud control: enable online configuration to add scenarios, reducing unnecessary storage and bandwidth while meeting diverse data needs.

Professionalism: capture both basic device info and advanced runtime data such as classes, methods, libraries, and instructions.

Comprehensiveness: fully reflect the current device state.

Granularity: abstract and precisely split data from overview to atomic level.

Data cleaning: transform heterogeneous raw logs into readable, analysis‑ready formats.

Extensibility: allow easy addition of unsupported tracking data with minimal effort.

3. Smart Data Tracking Solution

3.1 Basic Data Flow

Event : entry point for data collection.

Filter : flexible filtering resolves the tension between coverage and performance.

Collect : abstract and categorize data, exposing collection functions for extensibility.

Process : format, extract, transform, merge, split, and simplify collected data.

Send : encrypt, package, and transmit data according to a predefined protocol.

3.2 Overall Framework

3.3 Main Functional Modules

Event Ingestion : custom events and categorized events (network, system, UI) are unified for low integration cost.

Event information includes type, name, timestamp, and data, with format varying by event type.

Filters: two‑layer filtering; first layer discards events, second layer passes events for reporting while caching partially filtered events for later traceability.

Condition Filters: match conditions using operators (e.g., equals, greater than, contains, regex) across Number, String, and Array inputs.

Counters and Timers

Functions: data collection and processing are expressed as functions handling inputs and outputs of common types (Number, String, Array, Map) or void.

3.4 Communication Protocol

Modules communicate using standard data formats such as JSON or Protobuf.

4. Usage Example

Example: add user location information when a user places an order (e.g., https://x.huolala.cn/user_order?uid=1000&args=xxx).

Filter: match network event type = 1.

Filter: match request_host == x.huolala.cn && request_path == user_order.

Add location data to the event.

Encrypt, package, and report the data.

5. Problems and Challenges

Fully integrating all event types poses performance challenges; improper handling can lead to UI lag or other extreme issues.