How AirTrack Enables Real‑Time A/B Testing for Mobile Apps

This article is a summary of the March 17 sharing session of the “Mobile Frontline” community, presenting AirTrack – a trial‑and‑error platform built for mobile A/B testing and gray‑release.

A/B Testing

A/B testing centers on bucket allocation, executing different logic based on bucket results to find the optimal solution. It is used mainly for comparison experiments and gray releases.

System Overview

A single A/B testing framework is insufficient; surrounding capabilities are required. AirTrack integrates device/user data collection, native dynamic logic, condition distribution, and a data collection & analysis platform.

AirTrack consists of three core parts: dynamic experiment conditions, a real‑time calculation SDK, and a data collection & feedback platform.

Dynamic Experiment Conditions

Unlike web A/B testing where the backend handles condition calculation, AirTrack moves the computation to the app side, eliminating reliance on data APIs. Experiment conditions are expressed as a tree: the root is the computation start, non‑leaf nodes are logical expressions, and leaf nodes hold the final result.

The condition tree is described in JSON and pushed to the app via the configuration center, enabling dynamic updates.

Real‑time SDK

App code initializes an AirTrack instance, registers experiment names and associated logic. At runtime the SDK fetches conditions from the configuration center, evaluates the condition tree from the root, determines the leaf node (bucket), and executes the corresponding business logic.

Node Types in the Condition Tree

Attribute‑based operations – use values from the state center to evaluate logical expressions.

Hash operations – combine device ID, condition name, and node depth, hash to a 0‑10,000 range, and map to a bucket according to defined ratios.

Data Collection & Feedback Platform

The existing data collection SDK in the Taobao app is extended so that AirTrack automatically logs bucket information. Collected logs feed a page‑oriented analytics platform that shows PV, UV, click‑through rates, entry/exit flow, conversion rates, and bucket‑level statistics.

Practical Applications

Configuration Gray Release : Define gray‑release plans as A/B experiment conditions (e.g., 10% traffic for the first 20 minutes, 50% for the next 20 minutes, etc.) and roll out gradually while monitoring metrics.

Homepage Module Comparison : Split 20% of traffic, run old and new logic side‑by‑side for two days, compare conversion and display metrics, and iterate until the new version meets expectations.

Personalized Pop‑ups : Use AirTrack to assign users to buckets based on device and user attributes, enabling smart, context‑aware pop‑up displays instead of simple time‑based triggers.

Q&A Highlights

Q: Is the logic dynamically loaded? A: Only the layout is dynamic; the core logic is still static, though research is ongoing to make native logic dynamic via JavaScript.

Q: Why not just push a JSON config from the backend? A: Because control of ratios and real‑time adjustments must be reusable across multiple business lines, not tied to a single backend.

Q: What does “bucket” mean? A: It denotes the segment a user belongs to; the leaf node of the condition tree determines the bucket.

Q: How is percentage control achieved? A: By hashing device ID with experiment name.

Q: Can hard‑coded logic be A/B tested? A: Yes; hard‑coded UI elements (e.g., search box color) can be varied by registering multiple values in the SDK and switching via configuration.

Q: How are users bucketed on mobile? A: The condition tree JSON is sent to the device; the SDK traverses from root to leaf to determine the bucket.

Overall, AirTrack demonstrates how a code‑level A/B testing framework combined with dynamic configuration can accelerate decision‑making, reduce risk, and enable personalized experiences in mobile applications.