Intelligent Gray Release Data Analysis System Practice for Game Center

This article introduces vivo Game Center's practical experience in building a gray release (canary release) data analysis system, providing a comprehensive intelligent gray data solution across four dimensions: experimental thinking, mathematical methods, data models, and product solutions. The solution ensures scientific version evaluation, project progress, and rapid closed-loop of gray verification.

The gaming business has large user scale, long business chains, and complex data logic. As the core user product of the gaming business platform, Game Center requires small-scale gray verification before each version release. Since 2021, important versions begin gray testing every 1-2 weeks on average, with multiple versions sometimes undergoing simultaneous gray testing.

The gray process involves three key data questions: how to ensure scientific gray evaluation, how to improve gray data output efficiency to guarantee project progress, and how to quickly locate problems when indicator anomalies occur.

The solution's highlights include the introduction of indicator anomaly root cause analysis methods and the design of fully automated product solutions. The system incorporates hypothesis testing for statistical significance, sample historical difference verification using 7-day sliding window sampling, indicator logic analysis method, and the Adtributor algorithm for multi-dimensional time series anomaly root cause analysis.

The implementation includes sample size estimation with multiple confidence levels, significance testing for rate-based and mean-based indicators, automated negative indicator root cause analysis with four steps (anomaly detection, sample historical difference verification, indicator logic decomposition, and Adtributor automatic root cause analysis), and intelligent gray report generation with automatic version information retrieval and conclusion presentation.

Code implementation for significance testing:

# Rate-based indicators

variation_visitors
# Gray version indicator denominator

control_visitors
# Control version indicator denominator

variation_p
# Gray version indicator value

control_p
# Control version indicator value

z
# z-value at different confidence levels (90%/95%/99%)

# Calculate indicator standard deviation

variation_se = math.sqrt(variation_p * (
1
- variation_p))

control_se = math.sqrt(control_p * (
1
- control_p))

# Calculate indicator change value and rate

gap = variation_p - control_p

rate = variation_p / control_p -
1

# Calculate confidence interval

gap_interval_sdown = gap - z * math.sqrt(math.pow(control_se,
2
) / control_visitors + math.pow(variation_se,
2
) / variation_visitors)

gap_interval_sup = gap + z * math.sqrt(math.pow(control_se,
2
) / control_visitors + math.pow(variation_se,
2
) / variation_visitors)

confidence_interval_sdown = gap_interval_sdown / control_p

confidence_interval_sup = gap_interval_sup / control_p

# Significance judgment

if
(confidence_interval_sdown >
0
and
confidence_interval_sup >
0
)
or
(confidence_interval_sdown <
0
and
confidence_interval_sup <
0
):

print(
"Significant"
)

elif
(confidence_interval_sdown >
0
and
confidence_interval_sup <
0
)
or
(confidence_interval_sdown <
0
and
confidence_interval_sup >
0
):

print(
"Not Significant"
)