Ctrip and Qunar ABTest Promotion Week: Expert Insights on Experimentation and Big Data Practices

From August 26 to September 1, Ctrip Technology Center jointly with the Basic Business R&D Department held an internal ABTest Promotion Week. The event invited industry experts to deliver eight专题分享 covering ABTest basics, the most popular architectures and algorithms, future development trends, and the importance of ABTest in the development of big‑data products.

The opening guest was Tang Juan, Director of Qunar Platform Business Department, who shared Qunar's ABTest system evolution from 0 to 1. In early 2016 Qunar and Ctrip's Basic Business R&D started cooperation, completing development and testing in just over two months, and launching the first ABTest experiment successfully.

Qunar's ABTest architecture consists of three major parts: a visual configuration system; a unified SDK (ReactNative + Hybrid + Android + iOS); and a reporting system. The system is already used in seven business lines, with more than 50 experiments conducted.

Tang emphasized that the system’s rollout has a positive impact on refined operations and that it will continue to be promoted across more business units, with ongoing functional expansions to meet diverse product needs.

Eric Ye, CTO of Ctrip, highlighted the three most important characteristics of big data—volume, velocity, and user‑centricity—then reviewed Ctrip’s big‑data development path, especially how the ABTest culture was formed and embedded in every employee.

In 2013, the first web user‑behavior data (UBT) was deployed, marking the start of Ctrip’s big‑data journey, and the first ABTest experiment went live the same year. Eric noted that a common mistake in ABTest experiments is lacking a big‑data mindset, such as ignoring results that do not show improvement.

With the rapid growth of mobile internet in 2014, Ctrip’s daily UBT data exceeded 50 TB, the number of ABTest experiments surpassed 1,400, and over 100 experiments run concurrently each day, positioning Ctrip as a domestic leader in ABTest.

The third speaker, Fan Cong, Data Director of Dazhong Dianping, shared their ABTest practice: why they introduced ABTest, the architecture, and optimization paths. He described how they designed a traffic‑layer model to solve flow issues of traditional AB experiments, integrated with search‑ad systems, dashboards, and provided auxiliary features such as experiment cloning and confidence tools.

The fourth guest, Gao Long, ABTest expert from Ctrip’s Vacation Business Unit, described how ABTest has been used since 2016 to select homepage recommendation algorithms and redesign the booking flow. He also shared rapid integration methods, problem‑handling techniques, and the process of generating ABTest reports.

The final speaker, He Shubo, senior ABTest algorithm expert from Ctrip’s Basic Business R&D Department, identified three main challenges of the ABTest system: handling non‑normal metric distributions (t‑test or Lift validity), ensuring result reproducibility, and balancing scientific rigor with commercial demands. He proposed solutions such as using non‑parametric runs tests, Bayesian methods, and BanditABTest to address these issues.