Implementing A/B Testing for Low‑Frequency Mobile App Home UI with a Hybrid Client‑Server Approach

In this article, the author, a Qunar mobile client engineer, discusses the challenges of conducting A/B experiments on low‑frequency home‑page UI features in a mobile app and proposes a hybrid client‑server solution.

The main issues include the need to avoid blocking app startup for server‑side experiment data, handling infrequent user sessions, and providing mechanisms to adjust, pause, or close experiments dynamically.

The implementation consists of two steps. First, each user is assigned a unique identifier; after evaluating options such as IMEI, Android ID, and UUID, the team chose UUID despite its trade‑offs.

Second, the app creates logical groups (e.g., b0‑b99) using a hash function like MurmurHash, then maps these logical groups to business groups (e.g., B0‑B3) according to the desired traffic split. The mapping is cached on the client.

An example of the business‑to‑logical group mapping is shown in the table below.

Business Group

Logical Group

B0

b0~b9

B1

b10~b19

B2

b20~b29

B3

b30~b99

When the app requests experiment data from the server, it sends both the business and logical group information in a JSON payload, for example:

{
  "init": {
    "bucket": "[b0|b1...]",
    "time": "xxx"
  },
  "current": {
    "bucket": "[B0|B1...]",
    "time": "xxx"
  }
}

The server only needs to modify the mapping if the experiment configuration changes; otherwise no action is required. To shift a portion of users from one business group to another, the server updates the logical‑to‑business mapping and pushes the new JSON to the client, e.g.:

{
  "bucket": "A|B|C...",
  "time": "xxx"
}

Frequent changes to user groups are discouraged because they can bias results. When moving to a second experimental phase with a different traffic split, the cached client data must be cleared and the groups recomputed, as illustrated by the updated tables below.

Business Group

Logical Group

B0

b0~b9,b30~b39

B1

b10~b19,b40~b49

B2

b20~b29,b50~b59

B3

b60~b99

Finally, the author recommends using a unified bucket experiment to manage multiple concurrent experiments and ensure an even distribution of users across groups.