Cloud‑Native and Intelligent Fusion: Key Trends Shaping the Future of Big Data

On November 30, 2022, Tencent announced the release of the IDC 2022 Cloud Product Evolution Trend Whitepaper, highlighting big data as a key track and identifying four trends: cloud‑native, data governance, intelligent fusion, and privacy computing.

The author then reflects on why cloud‑native and intelligent fusion are the driving forces behind big data’s development.

Big data technology traces back to Google’s three seminal papers—Google File System, MapReduce, and BigTable—which formed the foundation for massive storage and analysis. The open‑source community built the Hadoop ecosystem as a faithful implementation of Google’s stack.

Early Hadoop deployments required hundreds of physical machines, high electricity costs, and a dedicated operations team, making entry barriers steep. Cloud computing initially offered only basic IaaS services (virtual machines, object storage, networking) and did not directly address big data workloads.

When enterprises began running Hadoop clusters on public‑cloud VMs (initially only on Amazon), they realized the cost of keeping a full cluster running 24/7 was prohibitive. The breakthrough came with using object storage as a thin API layer for HDFS, enabling compute‑storage separation: clusters are spun up only when computation is needed, and idle periods rely on cheap object storage.

This architecture, now standard in many cloud big‑data services, led to managed MapReduce offerings (e.g., Tencent Cloud Elastic MapReduce). While compute‑storage separation is common, hybrid designs that combine traditional tightly‑coupled storage‑compute for performance‑critical workloads with separated architectures for elasticity are also used.

Beyond storage, container technologies have become the foundation of cloud‑native big data, providing elastic scaling of resources that physical clusters cannot achieve.

The convergence of big data and artificial intelligence is highlighted: deep learning requires massive compute and large training datasets, making the synergy between AI and big data essential for intelligent decision‑making.

To make these capabilities accessible to smaller enterprises, cloud platforms lower the barrier to entry, offering SaaS‑style big‑data services. Tencent’s Cloud Big Data Intelligent Recommendation Platform exemplifies this by combining massive data, AI algorithms, and easy‑to‑use APIs, delivering 10‑20× KPI improvements for a leading domestic sports brand’s WeChat mini‑program.

In summary, embracing cloud‑native principles—compute‑storage separation, containerization, managed services—and integrating AI are crucial for big data to overcome high costs and operational complexity, enabling broader adoption across industries.