Open Source Practices and Trends in Big Data and AI – Insights from the Tencent Cloud Developer Conference

On December 15, Tencent Cloud hosted the inaugural "Tencent Cloud+ Community Developer Conference" in Beijing. The event, themed "New Trends • New Technologies • New Applications," gathered more than 40 technical experts and over 1,000 developers to explore the latest developments in artificial intelligence, big data, IoT, mini‑programs, and operations development.

The big‑data & AI track was organized into three parts: (1) an open‑source view of technology, products, and ecosystems; (2) a review of Tencent Cloud’s big‑data practice and recent contributions; (3) a discussion of hot topics and future trends in the open‑source big‑data and AI ecosystem.

Open‑source: Technology, Products, and Ecosystem

The recent merger of Cloudera and Hortonworks, two historically competing big‑data vendors, sparked debate about whether the move represents a strategic alliance, a response to market pressure, or a shift toward partial‑open‑source licensing. Some vendors have also altered their licenses (e.g., MongoDB, Redis), imposing new restrictions that limit commercial use.

These changes illustrate an "iceberg dilemma" for open‑source products: users and enterprises often focus on surface‑level features, performance, and price, while overlooking the complex, less visible layers of maturity, security, and maintainability beneath the surface. Ignoring these hidden layers can erode the stability of the ecosystem.

From a lifecycle perspective, open‑source projects benefit from rapid development but still require rigorous testing and maintenance. Without proper testing, open‑source solutions may appear unstable, which is a perception issue rather than a flaw in the code itself.

Tencent Cloud Big‑Data Open‑Source Practice and Contributions

The Tencent Cloud Data Warehouse product "Sparkling" builds on Hadoop, Spark, and other open‑source components. It integrates multiple data sources (COS, cloud databases, elastic MapReduce, relational databases) and provides a development IDE supporting SQL, Python, and R. Optimizations made by Tencent Cloud have been contributed back to the community as patches, including enhancements to SparkSQL, Parquet bloom‑filter support, and column‑store MVCC/ACID features.

Tencent Cloud is a platinum sponsor of the Apache Software Foundation and actively participates in projects such as Hadoop Ozone and Spark Hydrogen, helping to coordinate releases and improve community health.

Hot Open‑Source Technologies in Big Data & AI

Apache Hadoop, now over a decade old, continues to evolve with cloud‑native integrations and AI platform collaborations. The convergence of big data and AI is evident in projects like Spark’s "Hydrogen" plan, which aims to unify data processing and AI workloads on a distributed scheduler. Deep learning benefits from massive datasets, and Tencent’s open‑source Angel framework exemplifies this synergy.

Overall, the speaker emphasizes that open‑source projects are public resources—like water—that require community stewardship. Sustainable development depends on investing in both the visible surface and the hidden foundation of the ecosystem.

Conclusion

Open‑source should be valued for its community and people, not merely for the code. Developers are encouraged to contribute boldly, and users should adopt open‑source solutions confidently, as they can meet most scenarios without being locked into proprietary software.