Building a Scalable Distributed Cron: Google‑Level Design Simplified for Startups

Reference Papers

Distributed Periodic Scheduling with Cron

Reliable Cron across the Planet

Key Requirements Derived from the Papers

High‑availability strategy for scheduled jobs, including fault tolerance, idempotency, execution monitoring, and handling of node failures.

Deployment considerations for massive cron management, such as cross‑host migration, failure compensation, and state preservation.

Preservation of intermediate data and environment dependencies during cron‑job migration.

Isolation and resource‑aware scheduling to guarantee real‑time execution for short‑interval jobs.

Google Distributed Cron Design

State storage is implemented as an internal sub‑service rather than external file systems (e.g., GFS, HDFS) to reduce dependencies and support high‑frequency small‑file access.

Fast Paxos is used to provide decentralized high availability and rapid leader election.

All nodes store task allocation and scheduling information, but only the elected leader executes jobs. When the leader fails, a new election occurs and the new leader inherits persisted state via RPC with the scheduling center.

Leader‑driven execution is synchronized through Paxos; only the leader communicates with the central scheduler.

If a follower takes over a partially‑executed job, configurable policies decide whether to retry, skip, or risk duplicate execution, while also mitigating thundering‑herd effects.

Simplified Startup Implementation

For most startups the full Google‑scale solution is unnecessary. The following lightweight design replaces Fast Paxos with the open‑source Raft algorithm and uses etcd for both state storage and leader election.

Implementation details:

All nodes read the current job state from etcd. The elected etcd leader becomes the cron scheduler.

The leader persists job definitions, assigns jobs to workers, handles job migration, and performs failover.

If a worker node crashes, the leader re‑balances its jobs to other healthy nodes.

If the leader itself fails, Raft triggers a new election; the new leader immediately resumes job distribution using the persisted state.

State inspection and debugging are performed via etcd logs and versioned data.

Additional Features Added for the Recommendation System Team

Cron job specification and management UI.

Result visualization, execution monitoring, and alerting.

Parent‑child relationships, mutual exclusion, concurrency limits, and simple orchestration.

Support for second‑level granularity and permanently looping tasks.

Planned integration with containerized environments for unified monitoring.

Architecture Diagram

Operational Views

Task list and execution status UI:

Node‑level monitoring view:

Online log viewer for job execution: