Architecture Evolution and Implementation of the Intelligent Acceleration Engine in the 58 Big Data Platform

The Intelligent Acceleration Engine is a self‑developed complex computing component of the 58 Big Data Platform, crucial for supporting business growth and platform stability; with the maturation of big‑data technologies and rapid AIGC development, the team seeks a technical iteration and architectural upgrade to achieve significant cost reduction and efficiency gains.

Architecture analysis reveals the original engine’s limitations: high code coupling across modules, difficulty extending to multiple engines, stability challenges as a combined gateway and compute service, and cross‑datacenter isolation needs. (See Figure 1 and Figure 2 for the original architecture.)

To address these issues, the architecture is upgraded by introducing Apache Kyuubi as an independent gateway service and adding StarRocks as a diversified compute engine, thereby improving query performance and solving existing production problems. (See Figure 3 for the new architecture.)

Capability building includes a multi‑tenant redesign that replaces proxy‑user authentication with a doAs session‑level execution model, group‑based isolation for multi‑engine support, and enhanced cross‑datacenter scheduling that distributes SQL based on account and data volume. Engine selection strategies are enriched with RANDOM, LEASTACTIVE, and WEIGHT policies, while parsing and forwarding are strengthened through SQL Parse, SQLGlot dialect rewriting, HBO‑based historical optimization, and an AI Matrix for algorithm‑driven plan selection. (Figures 4‑8 illustrate these components.)

Implementation work covers extensive compatibility adaptations across syntax parsing, metadata binding, query optimization, and execution phases—modifying StarRocks to accept Spark syntax, integrating SQLGlot, extending storage formats, and providing Java UDF support via HDFS. Stability fixes target CBO‑generated massive SQL statements and FE memory overloads. Usability improvements add persistent SQL blacklist storage and containerized deployment with mixed cloud‑on‑premise resources. (Figures 9‑15 show the detailed processes and results.)

Landing performance shows daily SQL executions exceeding 100 k, StarRocks‑based lake‑warehouse queries surpassing 60 k, ETL instances over 10 k with average efficiency gains of 41 %, HiveSQL migration rate of 92.4 % and P95 latency improvement of 43.8 %, AI algorithm accuracy increase of 82 % and failover reduction of 50 %, and overall CPU resource savings of more than 15 k cores.

The summary outlook emphasizes continued iteration of Kyuubi and StarRocks, deeper exploration of Spark 3.5, vectorization research, and AI‑driven innovations to further advance data intelligence and support user‑centric, technology‑driven growth.