Comprehensive Overview and Source Code Analysis of NetEase Spark Kyuubi

Kyuubi is an open‑source high‑performance general‑purpose JDBC and SQL execution engine developed by NetEase's Data Lake team, built on top of Apache Spark to address the multi‑tenant, resource isolation, and high‑availability limitations of Spark ThriftServer.

The article first explains Spark ThriftServer's design, its reliance on Apache Thrift for RPC communication, and its shortcomings such as lack of multi‑tenant isolation and single‑point failure, motivating the need for Kyuubi.

Kyuubi's architecture is divided into four layers: User Layer, Service Discovery Layer (using Zookeeper), Kyuubi Server Layer, and Kyuubi Engine Layer. The Server layer receives user requests via RPC, forwards them to the Engine layer, which holds a SparkSession per user to execute SQL.

Key components include FrontendService (the RPC server built on TThreadPoolServer ), KyuubiServiceDiscovery (Zookeeper client for service registration), KyuubiSessionManager and KyuubiOperationManager (session and operation handling), and their counterparts in the Engine layer ( SparkSQLSessionManager , SparkSQLOperationManager ).

Startup processes are described in detail: KyuubiServer initializes configuration, creates a TThreadPoolServer , registers itself in Zookeeper under /kyuubi , and starts background services; SparkSQLEngine similarly starts, registers under a user‑specific Zookeeper namespace (e.g., /kyuubi_USER/xpleaf ), and listens on a dynamically assigned port.

Session establishment involves the Server checking Zookeeper for an existing Engine instance for the user; if none exists, it launches a SparkSQLEngine via spark-submit with appropriate configurations. The Server then creates an RPC client to the Engine, opens a remote session, and caches the session handles.

SQL execution is asynchronous: the Server receives an ExecuteStatement request, creates a Kyuubi ExecuteStatement operation, forwards the statement to the Engine via RPC, and tracks the remote operation handle. The Engine runs the statement using spark.sql(statement) , stores results in an iterator, and updates operation state. FetchResults requests retrieve results from the iterator after confirming the operation has finished.

Throughout the article, numerous code snippets illustrate critical classes and methods, such as the Thrift service implementation, Zookeeper node creation, and Spark session initialization, providing readers with concrete references for deeper exploration or customization.