Ctrip Data Platform 2.0 Architecture and Evolution: Multi‑IDC Storage, Tiered Data, Scheduling, and Spark/Kyuubi Enhancements

The Ctrip Data Platform 1.0, built on HDFS, YARN, Spark and Hive, served the company from 2017 to 2022. Rapid data growth in 2023 (several PB per day) forced the construction of a third IDC and a redesign of the platform.

Key challenges included supporting a multi‑IDC architecture, relieving storage pressure by migrating cold data to object storage, expanding compute resources for both offline and online workloads, and upgrading Spark2 to Spark3.

Overall architecture (Data Platform 2.0) introduces a multi‑IDC storage layer with hot, warm, and cold tiers, transparent data migration, and read‑caching via Alluxio. The scheduling layer adds flexible priority scheduling and NodeManager mixing, while the engine layer upgrades to Spark3 and adopts Kyuubi as the unified query entry point.

Storage upgrades provide multi‑IDC data locality, tiered storage (hot on private cloud, warm on erasure‑coded nodes, cold archived to cloud object storage), and transparent migration tools such as Balancer, Mover, and Disk Balancer. HDFS Router‑based Federation is used for namespace splitting and cross‑cluster migrations, with FastCopy (based on Facebook’s FastCopy) enabling metadata‑only moves.

Read acceleration is achieved by integrating Alluxio, which offers transparent URI access, automatic master selection across IDC, and multi‑tenant caching without changing data locations.

Scheduling improvements include priority‑based YARN scheduling for P0/P1 tasks, NodeManager mixing (temporarily shutting down NodeManagers during low‑load periods and replacing them with Presto/Trino/StarRocks nodes), and offline/online mixed clusters using YARN node labels. A remote shuffle service (Celeborn) replaces the traditional External Shuffle Service, using push‑style shuffle, I/O aggregation, and a two‑replica mechanism to reduce fetch failures and improve performance.

Compute engine evolution covers the smooth migration from Spark2 to Spark3, leveraging Kyuubi’s plan‑only mode for SQL replay, extending BasicWriteTaskStats, handling Hive view compatibility, avoiding full UDF loading, and fixing zero‑size ORC file issues. Partition pruning is accelerated by a fast fallback that first fetches partition names then filters on the client, reducing pruning time from minutes to seconds. Data skew detection is added via JoinKeyRecorder and integrated into a diagnostic platform that generates lineage reports.

Kyuubi integration replaces the Spark2 Thrift Server, offering better isolation, multi‑tenant support, HA via Zookeeper, and dynamic engine lifecycle management. Kyuubi’s two‑layer architecture (Server + Engine) enables per‑user resource queues, graceful stop, and remote configuration. Full‑link lineage tracking is implemented by capturing session, operation, and YARN application IDs at the Spark SQL execution level and propagating them to HDFS audit logs.

Benefits of the 2.0 redesign include scalable architecture for future growth, tiered storage reducing costs, priority scheduling ensuring SLA for critical jobs, a 40% speedup after Spark3 migration, over 300k daily queries via Kyuubi, 30‑50% read‑speed improvement with Alluxio, and flexible mixed‑mode compute enabled by Celeborn.