How to Build a Scalable Distributed Task Scheduler from Scratch

1. Requirement

As the company’s business grows rapidly, more scripts need periodic execution. Using Linux's built‑in crontab for scheduling has become problematic: lack of unified management, inability to control or pause tasks, and no distinction between serial and parallel jobs.

Open‑source Quartz was considered, but it cannot meet our needs because it is single‑machine, has limited scalability, and runs tasks in the same process, risking the scheduler’s stability.

Desired Scheduler Features

Distributed: Use multiple machines, assign tasks to specific machine groups, and easily scale by adding nodes.

Flexible Control: Allow pause, resume, stop, and enforce concurrency limits; ensure serial tasks do not run in parallel.

Comprehensive Statistics: Real‑time view of tasks per machine, execution counts, concurrency, failure types, and execution time.

Stable and Robust: Scheduler must remain unaffected by task failures or exceptions.

The solution is to build a custom task scheduling system from the ground up.

2. System Composition

The scheduler is split into three subsystems:

Trigger System: Selects tasks based on defined strategies and executes them in separate subprocesses.

Monitor System: Continuously checks the health of trigger nodes; if a node crashes, it marks its running tasks as crashed and updates status and statistics.

Management System: A web UI that lets users control tasks, view results, and manage configurations.

Each subsystem runs as an independent distributed program, communicating via MySQL and using Redis for distributed locks. The architecture diagram is shown below:

Users upload task JARs to the same machine as the trigger system; the monitor and management systems can be placed elsewhere but should be deployed on multiple nodes to avoid single points of failure. After uploading, users configure cron expressions and constraints via the management UI, which stores the configuration in the database. The trigger system polls the database, schedules eligible tasks, and records execution details. The monitor system watches execution and raises alerts on anomalies.

3. Key Technical Points

Isolating Task Execution

Tasks run in separate subprocesses, preventing their code from compromising the scheduler.

Problem: Subprocess output can block if the parent does not read it.

Solution: Close default I/O streams in the parent and spawn a thread to read the subprocess error stream and log it.

Task Selection Strategy

The parent process selects tasks based on conditions such as next execution time, remaining run count, end time, concurrency limits, pause status, priority ordering, and available capacity.

Control Signals (Pause/Resume)

Management commands are written to the database; the trigger’s execution module polls for these signals. Pausing requires cooperation from the task code via a predefined interface that periodically checks for the pause flag.

Crash Detection

Trigger nodes send heartbeat updates to a database field. The monitor system checks the timestamp; if it hasn’t been updated within a threshold, the node is considered crashed and cleanup is performed.

Log Aggregation

Instead of creating a file per execution, logs are rolled by date, with each entry tagged by a unique execution ID, task ID, and thread ID, enabling efficient querying.

Dedicated Redis Instance

Separate Redis resources for the scheduler prevent task‑induced connection exhaustion from affecting the scheduler’s lock mechanism.

Exponential Back‑off for Lock Acquisition

When many tasks contend for a distributed lock, a gradient waiting strategy doubles the wait time after each failure, reducing lock contention under high concurrency.

Conclusion

The custom scheduler is now deployed across multiple domestic and overseas servers, handling close to a hundred concurrent tasks and over five million executions, fully replacing the inefficient crontab approach. Initial issues with network instability causing DB/Redis failures were mitigated with retries and back‑off. After five months of operation, the system remains stable with no downtime.