Apache Linkis Graduates to Top-Level Project – Overview, Core Features, Roadmap, and Ecosystem
The article announces Apache Linkis’s graduation to an Apache top‑level project, explains its role as a computing middleware linking applications to engines like Spark, Hive, and Flink, details its core capabilities, roadmap, ecosystem integrations, and provides official resources for the community.
On December 3, 2022 the Apache Software Foundation incubator approved the graduation of the Apache Linkis computing middleware project, and on January 18, 2023 Apache Linkis officially became a Top‑Level Project (TLP) of the ASF.
Apache Linkis, originally open‑sourced by the WeBank big‑data platform team in July 2019 and donated to the ASF in August 2021, serves as a computation middleware layer between upper‑level applications and lower‑level engines. It offers standard REST, JDBC, and Shell interfaces, enabling applications to connect to engines such as MySQL, Spark, Hive, Trino, Flink, and more, while managing user resources like variables, scripts, functions, and files across applications.
Key capabilities include:
Support for a wide range of underlying compute and storage engines (Spark, Hive, Flink, Python, Shell, OpenLooKeng, Presto, Trino, ElasticSearch, JDBC, SeaTunnel, etc.).
Rich language support (SparkSQL, HiveQL, Python, Shell, PySpark, Scala, JSON, Java).
Powerful governance features (multi‑level tagging, task routing, load balancing, multi‑tenant, traffic and resource control).
Full‑stack compute‑storage engine architecture handling batch, interactive, streaming, and data‑lake tasks.
Unified context service for shared resources (JAR, ZIP, properties, result sets, variables, UDFs).
Unified material management for cross‑user and cross‑system sharing.
Unified data‑source management (Hive, ElasticSearch, MySQL, Kafka, MongoDB, etc.) with version control and metadata queries.
Error‑code catalog with troubleshooting guidance.
Since its open‑source launch, the Linkis community has grown to over 7,600 users, with more than 2,600 companies trialing the sandbox, over 110 enterprises in production, handling more than 400 PB of data and serving over 5,000 users across finance, telecom, manufacturing, and internet sectors.
During its incubation, the project released seven Apache versions (approximately one every two months), added four new PPMC members and 13 committers, reaching 127 contributors. The community now presents a roadmap inviting further contributions to evolve Linkis into a world‑leading computation middleware.
The Linkis ecosystem integrates with numerous open‑source projects covering data analysis, workflow, data exchange, data quality, machine learning, and various compute/storage engines, reducing development effort for upper‑level tools.
Official resources:
Website: https://linkis.apache.org/
Repository: https://github.com/apache/linkis
Mailing list subscription: [email protected]
The article concludes with a call for the community to continue contributing and expanding the Linkis ecosystem.
Signed-in readers can open the original source through BestHub's protected redirect.
This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactand we will review it promptly.
Big Data Technology & Architecture
Wang Zhiwu, a big data expert, dedicated to sharing big data technology.
How this landed with the community
Was this worth your time?
0 Comments
Thoughtful readers leave field notes, pushback, and hard-won operational detail here.