Design and Architecture of a Distributed Task Scheduling System for Database Automation

Terminology : Defines key concepts such as Task, Celery, Worker, Broker, Backend, TaskID, TaskName, TaskType, TaskWorker, TaskScheduler, TaskConsole, DistributeLock, and TaskQueue.

Background : Highlights resource conflicts in parallel task execution, the need for an efficient scheduler, and the limitations of the existing Celery‑based platform.

Requirements : Lists five goals—support all task scenarios, record full task lifecycle, flexible queue strategy, ensure idempotency, and guarantee high performance, scalability, and availability.

Task Classification : Differentiates tasks by blocking property (synchronous vs asynchronous) and repetition property (simple, timed, permanent, topology). Provides examples of each type.

Task States : Describes states Init, Create, Prepare, Postpose, Running, Pause, Success, Fail, Finish, None, and Rollback, with a flow diagram.

System Architecture : The platform consists of three major modules—TaskScheduler (records task info, dispatches tasks), TaskWorker (executes tasks), and TaskConsole (entry point). Each module is further broken down into sub‑components such as API, Producer, DB, LogDumper, Dispatcher, Checker, Queue, Scheduler, and the corresponding interfaces.

TaskScheduler Sub‑Architecture : Includes API, Producer (timed‑task generator), DB (task metadata), LogDumper (log‑driven task creation), Dispatcher (queues tasks), Checker (status verification), Queue (FIFO), and Scheduler (pulls from queue and triggers workers).

TaskWorker Sub‑Architecture : Implements a unified task interface; all tasks must support endpoints for create (POST), query (GET), status (GET with action=status), notify (PUT), run (PUT with action=running), pause/continue, prepare, rollback, change, clean, heartbeat, progress, and summary.

Common Task API Example :

GET /{version}/task/{task_type}?task_name={task_name}&task_type={task_type}&task_id={task_id}

All request bodies use application/json and include fields such as create_info (task_type, task_name, important_level, http_url, parent_task_id, control_task_id) and process_arg (start_time, deadline_second, lock_timeout, max_retry, task_status).

Task Execution Flow : Starts with state check, acquires lock, prepares resources, runs the task, handles success/failure, cleans up, triggers child/center tasks in topology, and finally releases the lock.

Asynchronous Callback Flow : After a task notifies completion, it re‑acquires the lock, checks result, performs cleanup, triggers dependent tasks, and releases the lock.

Trigger (Producer) Design : Describes cold‑standby and distributed ownership models, lock registration, heartbeat updates, failover handling, and ensures a unique trigger owner across the cluster.

Lock Strategies : Lists local lock, distributed lock (MySQL, Redis, Zookeeper, etcd), and distributed lock with watch. Discusses reliability concerns with Zookeeper‑based locks and provides mitigation rules (short expiry, avoid full‑task lock, prefer local lock when possible, pre‑execution state validation).

Conclusion : By abstracting task architecture and standardizing interfaces, the new system overcomes Celery’s limitations, delivering high‑performance, reliable, and extensible task scheduling for the automation platform.