How Tencent Built the ABetterChoice SaaS A/B Testing Platform for Global Games

A/B Test Overview

A/B testing, derived from double‑blind biomedical trials, randomly splits users into control and treatment groups to obtain statistically reliable decisions for product features and strategy.

ABetterChoice: Global SaaS Experiment Platform

Launched in 2022, ABetterChoice abstracts Tencent’s internal A/B testing capabilities into a SaaS product that satisfies overseas compliance, multi‑cloud deployment, and diverse data‑ownership requirements. It standardizes experiment SQL with a single StarRocks compute engine, improving service reusability. The platform is already integrated with Honor of Kings (overseas), PUBG Mobile, and Ubisoft’s The Division.

Transformation Background

Three tenant groups drive the redesign:

Tencent‑owned overseas games that prefer Tencent Cloud storage.

Independent overseas studios requiring GDPR/CCPA‑compliant public‑cloud storage (GCP, AWS).

Independent users who may use Snowflake, Databricks, BigQuery, Redshift, etc.

The original Kappa architecture (StarRocks compute‑storage‑integrated) suffered from:

Heavy multi‑table joins consuming excessive compute; inability to scale compute independently.

Local SSD storage becoming cost‑inefficient as data lifecycle grew.

Tight coupling with Tencent‑internal components and Tencent Cloud, limiting overseas support.

Architecture Renovation

The redesign follows two principles: a unified data foundation across major public clouds and a lake‑warehouse, compute‑separated architecture.

On Tencent Cloud, Tencent Cloud Big Data Service (TBDS) provides the data foundation for overseas games.

On public clouds, Databricks creates a unified data‑lake ingress supporting object storage (GCS/S3/Blob) and downstream engines (BigQuery, Redshift, Snowflake).

Lake‑Warehouse Construction

StarRocks 3.1+ adds native support for Delta Lake and Iceberg, enabling high‑performance queries on lake data without pre‑loading. A single StarRocks engine serves all experiment SQL, eliminating data duplication and simplifying service reuse.

Hot‑Cold Data Separation

Partition Cooling

Experiment data is stored on local SSD for the first 14 days to ensure low‑latency computation. Data older than 14 days is automatically cooled to object storage via lake‑based scheduling; metadata is kept in the lake. During the one‑hour cooling window, queries are blocked to guarantee result correctness.

Hot‑Cold Mixed Queries

When a query spans both hot (SSD) and cold (object storage) partitions, StarRocks executes a mixed BE+CN plan: each node performs a partial GROUP BY exp_id and aggregation, then a second aggregation merges the intermediate results. This reduces data exchange and cuts average query latency by ~80%.

Multi‑Tenant Isolation

Isolation is enforced at three layers:

Query Engine Layer: A shared public StarRocks cluster uses weighted query scheduling. Each tenant receives a dedicated database and user for fine‑grained permission control. High‑priority tenants can be assigned exclusive clusters for dedicated resources.

Data Lake Layer: Databricks Unity Catalog isolates metadata and SELECT permissions per tenant.

Object Storage Layer: Separate storage buckets per tenant, with regional isolation and IAM‑based access control.

Summary and Outlook

ABetterChoice, powered by StarRocks and a dual‑cloud lake architecture, is live on public clouds and has onboarded Tencent’s overseas games and independent studios. Future work includes:

Further performance tuning of multi‑dimensional ad‑hoc queries on Delta Lake.

Customizing lake‑warehouse compliance features to meet global data‑privacy regulations.