Fixing Multi‑Version, Multi‑Cluster and HA with Apache Kyuubi for Spark/Flink

Background

With the rapid growth of the instant‑consumption business, data volume has increased from dozens to thousands of nodes. To handle large‑scale data, Spark and Flink were introduced, followed by Livy and Flink SQL Gateway as access gateways. The overall architecture is shown below.

Business Pain Points

Since the second half of 2024, the instant‑consumption platform has upgraded its big‑data foundation: Spark to 3.5, Flink to 1.19, Hadoop to 3.3, and added Iceberg and Paimon. The upgrade is performed by creating new clusters and gradually migrating tasks.

Using Livy and Flink SQL Gateway as gateways introduces several issues:

High code‑maintenance cost: separate gateways for Spark and Flink require duplicated code.

No multi‑version support: each engine version needs its own Livy or Flink SQL Gateway instance, increasing deployment and coordination effort.

No multi‑cluster or elastic scheduling: new clusters cannot be automatically used for load‑balanced task placement.

Submitted tasks lack high availability: after a Livy restart, task state is not persisted, causing the application to think the task failed.

No traffic control: during peak submission windows (~9,000 tasks per 10 minutes), each Spark submission spawns a JVM that consumes ~2 CPU cores for ~10 seconds, leading to timeouts when CPU limits are exceeded.

Apache Kyuubi Overview

Apache Kyuubi is a distributed, multi‑tenant, enterprise‑grade data gateway that provides standardized HTTP/JDBC/ODBC interfaces to reduce the technical barrier of accessing big‑data engines and enables efficient resource sharing and isolation.

Key features:

Multi‑engine support: compatible with Spark, Flink, Trino, etc., for interactive analysis and batch ETL.

Multi‑version support: configure multiple engine versions via session profiles.

Multi‑tenant and resource isolation: isolate backend engine instances per connection or session.

Overall Architecture

After integrating Kyuubi with Spark and Flink, the architecture looks as follows (two separate Kyuubi deployments, one for Spark and one for Flink).

Implementation Details

1. Support Multiple Engine Versions

Challenge : Deploying four Livy/Flink SQL Gateway instances for four engine versions is unsustainable.

Solution : Use Kyuubi session profiles to set engine‑specific environment variables, eliminating the need for separate gateways.

kyuubi.session.conf.advisor=org.apache.kyuubi.session.FileSessionConfAdvisor

Configure the profile:

kyuubi.batchConf.spark.kyuubi.engineEnv.SPARK_HOME /opt/spark-3.5.3-bin-msxf-hadoop2</code><code>kyuubi.batchConf.spark.kyuubi.engineEnv.SPARK_CONF_DIR /opt/spark-3.5.3-bin-msxf-hadoop2/conf</code><code>kyuubi.batchConf.spark.kyuubi.engineEnv.HADOOP_CONF_DIR /opt/spark-3.5.3-bin-msxf-hadoop2/conf/yarn-conf

Submit a batch using the profile:

curl -v -X 'POST' \
  "http://kyuubi:20099/api/v1/batches" \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  --data '{
        "batchType":"SPARK",
        "conf": {
          "kyuubi.session.conf.profile": "test"
        }
      }'

2. Support Multiple Clusters

Challenge : After upgrading Hadoop clusters, Kyuubi could submit to the new cluster but could not query task status because YarnApplicationOperation only handled a single global YARN configuration.

Solution : Extend YarnApplicationOperation to initialize multiple YarnClients based on the session profile, and cache ApplicationIds to avoid expensive YARN tag scans.

3. Elastic Scheduling

Challenge : When one cluster is saturated while another has spare capacity, tasks should be routed to the less‑loaded cluster.

Solution : Add a load‑collector module in Kyuubi, expose a routing API, and allow dynamic policy definitions (random, round‑robin, weight, time‑window, load‑based) via Python scripts.

Scheduling flow:

4. Task High Availability

Challenge : Livy does not persist session or statement state, causing task failure after a restart.

Solution : Use Kyuubi Batch mode, which persists batch state and communicates only with YARN for status queries. A unified SQL‑execution JAR was developed for Spark batch jobs.

curl -v -X 'POST' \
  "http://kyuubi:20099/api/v1/batches" \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  --data '{
        "batchType":"SPARK",
        "resource":"hdfs://nameservice/resource/spark-sql-executor-1.0.jar",
        "className":"SparkSqlExecutor",
        "name":"SparkSqlExecutor",
        "conf": {
          "spark.msxf.dsp.task.sql": "c2VsZWN0ICogZnJvbSB0ZXN0X2RiLnRlc3RfdGFibGUK"
        }
      }'

5. Traffic Control

Challenge : Unbounded concurrent Spark submissions cause CPU saturation and request timeouts.

Solution : Limit concurrent Spark submissions per Kyuubi instance (default CPU cores / 2). Excess submissions are queued; tasks exceeding a timeout are marked failed and reported.

Monitoring during peak load shows each instance using ~8 of 12 CPU cores, meeting expectations.

Results

Support for over 180,000 Spark YARN applications per day.

Reduced gateway instances from eight to two, dramatically lowering SRE deployment and maintenance effort.

Elastic scheduling shifts tasks from saturated to idle clusters, saving roughly 10,000 CPU cores daily and shortening job runtimes.

Release cycle for Kyuubi versions shortened from days to hours, eliminating the need for application‑service coordination.

Future Plans

Implement high‑availability for Kyuubi Session mode to prevent SQL task failures after restarts.

Add Spark Connect support to simplify client‑side deployment for tools like Jupyter.

Contribute the developed common functionalities back to the open‑source community.