How Tencent Built Its Massive Big Data Platform Over a Decade

Introduction: Tencent, as one of China's largest internet companies, handles massive daily data volumes. Efficient storage, management, and utilization of this data are essential to avoid turning data assets into waste.

The big data platform is a core infrastructure that processes tens of millions of offline tasks and tens of trillions of real‑time computations each day to meet billions of analytical queries.

01 Tencent Big Data Construction Philosophy

When the project was initiated, the team debated whether to build everything in‑house or adopt open‑source solutions. Urgent business needs in 2009 (e.g., QQ Space's “Happy Farm”) demanded a rapid data‑warehouse build, leading the team to choose open source for its rich community resources and faster deployment.

Although the team originally used C/C++ for billing systems, the big‑data ecosystem primarily relied on Java, requiring a learning curve and recruitment of experienced developers. The ecosystem’s breadth also meant that each component needed extensive optimization to meet enterprise‑grade reliability.

02 Tencent Big Data Overall Architecture

Over more than a decade, Tencent’s big‑data platform has undergone three major architectural generations.

First generation (2009‑2011): Offline‑focused, built on Hadoop (TDW). Optimizations included expanding cluster scale, improving scheduling, enhancing disaster recovery, and integrating surrounding ecosystems such as an Oracle‑compatible SQL layer and PostgreSQL for small‑data analytics.

Second generation (2012‑2014): Added real‑time capabilities by integrating Spark for faster batch processing and Storm for millisecond‑level streaming. Built a real‑time data collection system (TDBank) that reduced ingestion latency from daily to seconds, and introduced resource and task scheduling that supported CPU, memory, network, and I/O dimensions.

Third generation (2015‑2019): Focused on AI and machine‑learning workloads. Developed the high‑performance distributed machine‑learning platform Angel in collaboration with Peking University, supporting billions‑scale models, data and model parallelism, and online training. Angel was open‑sourced in 2017 and later donated to the Linux Foundation.

The platform also incorporated heterogeneous resource management (GPU, FPGA) and tightly integrated a PostgreSQL‑based distributed database (TBase) to enable HTAP capabilities, bridging big data and transactional processing.

Today, Tencent continues to explore next‑generation directions such as batch‑stream fusion, cloud‑native big data, AI‑cloud convergence, data lakes, and privacy‑preserving computation, underscoring the foundational role of big data, AI, and cloud computing in its business ecosystem.