Big Data Trends and Future Directions – Insights from the Techo TVP Developer Summit Roundtable

On April 24, the Techo TVP Developer Summit held a round‑table discussion titled “Data’s Ice and Fire – From Online Database Technology to Massive Data Analytics.” The session was hosted by Wen Ming (CEO of Zhiliu Technology, Tencent Cloud TVP) and featured Chang Lei (CEO of Oushu Technology, Tencent Cloud TVP), Guo Wei (CTO of Analysys, Tencent Cloud TVP), Chen Long (Technical Lead of Tencent Cloud Elastic MapReduce) and Li Haixiang (Database Expert Engineer at Tencent Cloud).

1. Has big‑data technology reached a bottleneck? Chang Lei explained that the retirement of several Apache projects (e.g., Kafka, Hadoop, Flink) does not indicate a bottleneck. Projects retire because their development cycles end or they are superseded by newer tools, which is a normal evolution similar to a person’s growth.

2. What are the development trends for big data in the next 3‑5 years? Guo Wei emphasized a shift toward cloud‑native data platforms. While many components will continue to proliferate, they will become more integrated with application scenarios, such as ClickHouse solving “last‑mile” analytics. The trend is toward more specialized databases rather than a single monolithic platform.

3. How does a data lake address the shortcomings of traditional data warehouses, and what is its future? Chen Long defined a data lake as a centralized storage that can hold arbitrary volumes of structured or unstructured data without requiring ETL, enabling diverse analyses. He described two dominant product forms: (a) a Presto‑style query layer that connects to heterogeneous sources, and (b) a Table‑Format + OLAP engine that normalizes data into a unified format for downstream processing.

4. Balancing cloud vendors, open‑source communities, and commercial open‑source companies Li Haixiang noted Tencent’s strong open‑source commitment: many internal products (e.g., TDSQL) are open‑sourced, and Tencent contributes back to projects like MySQL. He also highlighted growing collaboration between cloud providers and open‑source communities, such as Tencent Cloud’s partnership with MongoDB.

5. Can China produce the next “Snowflake”? The panel identified three conditions: (1) robust cloud infrastructure (already in place with Tencent Cloud, Alibaba Cloud, etc.), (2) a market where only ~40 % of enterprises have migrated to the cloud, indicating large untapped demand, and (3) a deep talent pool cultivated by years of Hadoop and big‑data adoption.

6. Future technology interests Speakers highlighted real‑time data processing, cloud‑native storage systems for streaming data, GPU‑accelerated analytics, and the need for more efficient real‑time OLAP engines that can scale cost‑effectively.

7. Career advice for data professionals Li Haixiang suggested focusing on a single domain for a long time to build depth. Chen Long outlined three career paths: (a) kernel development (distributed storage/computation), (b) data architect (bridging cloud products and business needs), and (c) data scientist (leveraging machine‑learning on top of data platforms). Guo Wei added that future data engineers must be “dragon‑type” professionals—combining skills of engineers, analysts, product managers, and operators.

The discussion concluded with a recommendation to attend industry events like the Techo TVP summit to stay abreast of emerging technologies and to invest in foundational work, which offers long‑term value for both individuals and the broader data ecosystem.