Linkis: Open‑Source Big Data Middleware Joins the Apache Incubator

Linkis, an open‑source computing middleware developed by WeBank, has been accepted into the Apache Software Foundation Incubator.

It provides a layer between upper‑level applications and underlying engines, offering standard REST, WebSocket, and JDBC interfaces to connect to engines such as MySQL, Spark, Hive, Presto, Flink, etc., and enables sharing of variables, scripts, functions, and resource files across applications.

The middleware delivers strong connectivity, reuse, orchestration, extensibility, and governance capabilities, decoupling application and engine layers to simplify network calls, reduce complexity, and lower development and maintenance costs.

Since its open‑source release in 2019, Linkis has attracted over 700 trial enterprises and 1,000+ sandbox users across finance, telecom, manufacturing, and internet sectors, serving as a unified entry for big‑data platforms and a governance tool for compute requests.

Key features include support for numerous storage engines (Spark, Hive, Python, Presto, Elasticsearch, MLSQL, TiSpark, JDBC, Shell, with Flink and Impala in progress) and multiple script languages; advanced compute governance with multi‑level label routing, load balancing, multi‑tenant, traffic and resource control; full‑stack engine architecture; unified context and material services for resource sharing.

The architecture follows a micro‑service model divided into compute‑governance services, common enhancement services, and micro‑service governance services, leveraging Spring Cloud Gateway, Eureka, and OpenFeign.

Linkis aims to become a replacement for Zeppelin as a powerful tool for data platforms and mid‑platforms, with future plans to add full Flink support and expand its ecosystem.

Repository: https://github.com/WeBankFinTech/Linkis (to be migrated to Apache); mailing lists: [email protected], [email protected].