Building an AB Experiment System for User Growth Scenarios

The article introduces the need for a specialized AB experiment system in user growth (UG) scenarios, where new users are acquired through channels such as paid ads, ASO, and SEO, then guided through activation, maturation, and potential churn phases.

It explains the basic AB experiment principle: random traffic allocation to control and treatment groups, with two main splitting methods—platform‑based splitting that requires a stable device ID, and client‑side local splitting that can occur at device initialization but may suffer from bias.

Specific problems in new‑user contexts are highlighted, including the difficulty of achieving early splitting (up to 18.62% of users lack an ID at the moment of split), the high value of new‑user traffic, and survivorship bias caused by delayed metric collection.

To address these issues, a new experiment system is proposed, emphasizing three ID selection principles: compliance, timeliness (available at first launch), and uniqueness (stable within a single install cycle and one‑to‑one with metric IDs).

The scientific validity of the system is verified through uniformity and orthogonality tests of the splitting IDs, as well as statistical checks (t‑test type‑I error rates, normality of t‑statistics, uniformity of p‑values) that confirm the experiment’s randomness and reliability.

A practical UG case study demonstrates how the new system improves effective new‑device count (+1%) and retention rate, while also revealing that some metrics may show negative effects, underscoring the need for multi‑dimensional evaluation.

Finally, the article summarizes that existing UG experiment frameworks cannot fully address new‑user traffic challenges, and that the proposed system, with its rigorous ID selection and validation, enables a two‑dimensional optimization of device acquisition and retention, delivering measurable business gains.