Design and Architecture of the 58DP Big Data Platform Task Scheduling System

58DP is a general‑purpose distributed scheduling platform tightly integrated with the 58 big data ecosystem, offering one‑stop data development and scheduling services; it serves all business lines, supports ETL, Hive, Spark, Storm and other common big‑data components, handling over 100,000 tasks daily with high fault tolerance, easy scalability, and high concurrency.

The system architecture includes the Bakery UI, Duolong scheduling API, Signaler, and Bakery‑cron services. Bakery provides a web interface for data development, scheduling, and metadata management, supporting components such as Spark SQL/Streaming, Hive, Flink and custom tools, and integrates with HDFS and MySQL. Duolong abstracts the scheduling complexities through a simple API.

Technical deployment uses an active‑standby Bakery behind Nginx for traffic switching, Signaler as a micro‑service also behind Nginx, a separate Bakery‑cron sync service, and a Duolong cluster composed of master‑slave Scheduler nodes and multiple Worker nodes. Scheduler nodes acquire master status via Zookeeper locks, while Workers register with the Scheduler, send heartbeats, and receive tasks based on load‑balancing policies.

High‑availability is achieved through Zookeeper‑based master‑slave failover for the Scheduler, smooth stateful task transition, and automatic Worker reconnection. Workers are considered unavailable if no heartbeat is received within ten minutes, prompting task reclamation and redistribution. Task recovery relies on ack timeouts and execution timeouts.

Resource management uses a slot model where each Worker thread occupies a slot. The number of slots reflects a Worker's parallel processing capacity. Load balancing selects Workers with memory usage below 90% and the highest number of free slots, enabling dynamic task distribution and horizontal scaling.

The scheduling workflow starts with users creating tasks in the Bakery UI, after which the Scheduler dispatches them to Workers; periodic tasks are triggered by Quartz. Executors run specific job types and store logs in HBase, while a log‑sync service updates the Bakery UI in real time, completing the execution loop.

Task lifecycle consists of five stages: waiting for scheduling, rate‑limit queue, signal wait, serial queue, and slot wait. Tasks are prioritized into five categories (A‑E) with the order: re‑run > retry > system > API > manual. Queue polling first considers the number of wait cycles, then priority, ensuring high‑priority tasks are processed first.

Dependency handling uses a signal mechanism: downstream tasks execute only after all upstream signals are successfully generated. Failure recovery employs a failover retry mechanism with configurable retry counts, mitigating failures caused by platform instability.

Business solutions address complex scenarios such as historical data re‑run. Self‑dependency allows jobs to depend on the previous day's output, ensuring correct chronological execution. An elastic queue prevents deadlocks when fixed queues are full by moving ready tasks to a flexible execution queue. The system also supports range‑based and upstream/downstream re‑run, with configuration fields like job_id, parent_id, run_cycle, and offset.

In summary, 58DP already provides comprehensive functionality across the entire group, but future work will focus on a ring‑no‑center architecture for higher stability, lightweight Workers for unified offline scheduling, dynamic topology scheduling, and AI‑driven task analysis and prediction to achieve finer‑grained resource control.

Author: Yao Lianbin, Senior Data Development Engineer, TEG Data Product R&D Department.