How Alibaba’s Open‑Source Big Data Ecosystem Is Accelerating Like Moore’s Law

Open‑Source Big Data Technology’s “Moore’s Law” Acceleration

On November 5, at the Yunqi Conference Integrated Big Data & AI Summit, Wang Feng, Vice President of Alibaba’s Open‑Source Committee for Big Data, reviewed thirteen years of Alibaba’s open‑source big data development, highlighting a shift from user feedback to collaborative innovation.

Since 2009 Alibaba has deployed Hadoop and other open‑source big data tools at massive scale. After rigorous internal testing during events such as Double 11, the company launched real‑time open‑source big data services in 2015, fully migrated to the cloud, and offered the E‑MapReduce platform and Flink‑based real‑time computing as public cloud services. Alibaba also contributed the Celeborn shuffle service to the Apache Incubator and helped make Flink the de‑facto standard for real‑time computing, creating an open, diverse, modern, and intelligent ecosystem.

At the summit Wang highlighted four technical trends: cloud‑native, real‑time, data lake, and intelligence. Alibaba’s open‑source big data stack now runs on a fully cloud‑native architecture, delivering elastic scaling and pay‑as‑you‑go consumption. The combination of Flink SQL and Table Store enables end‑to‑end real‑time data‑warehouse pipelines with incremental consistency. A unified cloud‑native data lake architecture moves from integrated compute‑storage to separated layers, supporting diverse compute models and intelligent, secure lake management. New “intelligent operation brain” features such as automatic Flink job tuning and EMR Doctor diagnostics deepen the platform’s value.

The product matrix was upgraded with E‑MapReduce 2.0, offering three‑fold elastic optimization, thousand‑node scaling, and the ability to spin up a 100‑node data‑lake cluster in three minutes. Integrated with EMR, OSS, and DataWorks, the solution passed the China Academy of Information & Communications Technologies cloud‑native data‑lake test, retained full HDFS compatibility, and enhanced lake permissions and lifecycle management. Flink’s complex‑event processing capabilities now support real‑time risk control, marketing, and minute‑level job diagnostics, boosting resource efficiency by 30 %. Alibaba Cloud partnered with Elastic to launch a cloud‑native Serverless Elasticsearch, cutting costs by 53 %, and with Cloudera on a hybrid‑cloud CDP offering a unified on‑premise and cloud experience.

Alibaba’s open‑source big data community has contributed over ten top‑level projects, cultivated more than fifty committers and PMC members, and logged over 1.5 million lines of code. It has built an Apache Flink ecosystem covering Flink CDC, Flink SQL, Flink ML, Flink CEP, and Flink Table Store, while the Celeborn shuffle engine entered the Apache Incubator.

The 2022 Open‑Source Big Data Heat Report, jointly released by the OpenAtom Foundation, X‑lab, and Alibaba’s Open‑Source Committee, analyzed 102 active projects and identified a “Moore’s Law”‑like pattern: every 40 months the heat value of open‑source projects doubles, driving a new wave of technology updates. Over the past eight years five major heat jumps occurred, with diversification, integration, and cloud‑native architectures emerging as dominant trends. Alibaba’s Apache Flink topped the stream‑processing heat ranking, while DataX, Flink CDC, and Apache Celeborn also featured prominently.