Cloud‑Native Big Data Solutions for the Financial Industry: Architecture, Deployment, Scheduling, and Resource Management

The financial industry faces growing big‑data demands, including high‑throughput batch jobs and low‑latency online services, which strain traditional Hadoop clusters due to stability and resource‑efficiency problems.

Cloud‑native platforms offer containerized isolation, customizable networking/storage, and easier operations, enabling better multi‑tenant resource sharing and higher overall cluster utilization without adding hardware.

However, migrating thousands of existing Hadoop jobs to cloud‑native environments is challenging because legacy engines (Flink, Spark) lack native support, cloud schedulers miss queue and job concepts, and native schedulers have limited throughput for short‑lived tasks.

Volcano Engine provides two deployment models to bridge this gap: (1) Serverless YARN , a cloud‑native implementation of YARN that preserves YARN APIs, AM/ResourceManager logic, and adds plugins for jar localization, shuffle services, logging, and monitoring; and (2) Arcee Operator , a unified Kubernetes operator that defines a common job CRD, handles exception recovery, and abstracts the underlying scheduler to support advanced policies such as priority and gang scheduling.

For scheduling, the GRO Scheduler introduces a three‑layer resource management hierarchy: a global multi‑datacenter resource lake (ResLake), per‑cluster quota‑aware scheduling, and node‑level agents. It adds queue and job abstractions via custom resources, supports quota control (Min/Max), and implements strategies like priority, gang, and DRF scheduling, as well as pre‑emptive queue and intra‑queue resource reclamation.

The GRO Agent runs on each node to enhance isolation (CPU, memory, disk, network) and protect online service SLAs by monitoring resource usage and, when necessary, evicting low‑priority batch pods or triggering pod migration.

ResLake unifies resources across multiple data‑center clusters, providing virtual queues, global quota enforcement, and intelligent placement of jobs near their required storage, while also offering disaster‑recovery modes (migration, active‑active, high‑availability) and cross‑cluster job distribution.

Combined, Serverless YARN, Arcee Operator, GRO Scheduler/Agent, and ResLake enable seamless migration of financial big‑data workloads to cloud‑native environments, achieving higher resource utilization, lower operational costs, and reliable, low‑latency processing for both online and offline services.