Why A/B Testing Isn’t a Silver Bullet: Lessons from ByteDance’s Data‑Driven Journey

Data‑Driven Decision Making and A/B Testing

On April 20, ByteDance Vice President Yang Zhenyuan spoke at the Volcano Engine Technology Open Day, sharing insights on data‑driven product development and the role of A/B testing.

Choosing the Right Goal

Yang emphasized that a clear, measurable goal is essential. He argued that “user profile” and “usage time” are poor objectives because they are hard to quantify and may mislead product direction.

He illustrated how focusing solely on usage time can inflate metrics while degrading user experience, using a case study of a news‑recommendation product that increased average session length by retaining only high‑quality users.

Evaluating Methods

Three evaluation approaches were discussed:

Experience judgment – human intuition, widely used but prone to inconsistency and bias.

Non‑A/B data analysis – correlation analysis, which can confuse correlation with causation (e.g., chocolate consumption vs. Nobel laureates).

A/B testing – the most reliable causal method when properly designed.

Limitations of A/B Testing

Key challenges include:

Independence – groups must be isolated; otherwise, cross‑effects (e.g., driver allocation in ride‑hailing) distort results.

Confidence – low statistical significance (e.g., p‑value 0.75) makes conclusions unreliable.

Short‑ vs long‑term effects – short‑term metrics may hide eventual improvements or declines.

ByteDance’s Practice

Since its founding, ByteDance has embedded A/B testing in product growth, running over 1,500 new experiments daily across 400+ services, totaling more than 700,000 experiments.

Applications span product naming, UI tweaks, recommendation algorithms, ad optimization, and growth initiatives. The internal Libra platform, launched in 2016 and opened to external customers in 2019, powers this large‑scale experimentation.

Case Study: Douyin’s Name

Multiple candidate names were tested with equal budget and placement. “Douyin” ranked second in the A/B test but was chosen for its stronger brand fit, demonstrating that A/B results must be combined with human judgment.

Conclusion

While A/B testing is a powerful tool, it is not a universal solution. Effective decision making requires well‑defined, measurable goals, awareness of methodological limits, and complementary qualitative insights.