Mobile Baidu Gray Release Testing: Process, User Selection, and Effectiveness Metrics

Gray release is the final stage of testing, essential for validating product quality and strategy online; testers must promptly address issues discovered during this phase to prevent negative user experiences and churn.

In the Mobile Baidu (Handbook) project, gray release has become a standardized step that continuously enhances issue discovery, localization, and fine‑grained rollout, resulting in a noticeable decline in online crash rates.

The release process supports two modes: random rollout, which distributes the new version to a randomly selected user pool, and precision rollout, which targets users who are active with the target feature or have previously reported related problems, enabling more efficient problem detection.

The overall gray‑release architecture consists of several stages, as illustrated in the diagram below.

Using a wallet‑related feature as an example, three user groups are selected for the precise gray release: (1) daily active wallet users identified by the big‑data team, (2) active feedback users from the UFO feedback platform, and (3) high‑quality feedback users mined by UFO’s automated analysis, limited to about 60 CUIDs per round.

The release method continues the client‑wide rollout approach but adds functional segmentation: precise gray release targets the identified wallet users, while random gray release serves as a supplement, with separate channel IDs distinguishing the two streams.

Effect measurement shows that precise gray release improves upgrade rates and speed, increases the efficiency of user‑feedback problem resolution, and achieves over 91% crash detection (including top‑severity crashes such as OOM and null‑pointer errors, which reach 100% detection), saving roughly one person‑day of manual effort per iteration.