Why Scheduled Tasks Run Multiple Times in a Cluster – The Real Issue Isn’t Quartz but Poor Architecture

Problem

In a single‑machine deployment a Spring @Scheduled method such as:

@Scheduled(cron = "0 */5 * * * ?")
public void syncOrderStatus() {
    // sync order status
}

executes once every five minutes. After scaling to three servers the same method is started by each Spring container, so at the scheduled time all three instances execute the job simultaneously, producing duplicate orders, SMS messages, coupons and database rows.

Root cause

@Scheduled is a local scheduler; each JVM registers its own copy of the job. Therefore three JVMs each run one copy of the task.

Simple fix: add a distributed lock

Many teams add a Redis lock:

@Scheduled(cron = "0 */5 * * * ?")
public void syncOrderStatus() {
    String lockKey = "lock:job:syncOrderStatus";
    Boolean success = redisTemplate.opsForValue()
        .setIfAbsent(lockKey, "1", Duration.ofMinutes(4));
    if (!Boolean.TRUE.equals(success)) {
        return;
    }
    try {
        orderService.sync();
    } finally {
        redisTemplate.delete(lockKey);
    }
}

This ensures only the first instance that acquires the lock proceeds.

First pitfall – lock expiration before task ends

If a task normally takes 2 minutes but the lock expires after 5 minutes, a slowdown can stretch execution to 8 minutes. After 5 minutes the lock expires, another instance acquires it, and both nodes run the job concurrently. The solution is a watchdog that continuously extends the lock while the task runs, e.g. Redisson RLock with automatic lease renewal:

RLock lock = redissonClient.getLock("job:syncOrderStatus");
boolean locked = false;
try {
    locked = lock.tryLock();
    if (!locked) {
        return;
    }
    orderService.sync();
} finally {
    if (locked && lock.isHeldByCurrentThread()) {
        lock.unlock();
    }
}

Redisson’s watchdog renews the lease automatically when no fixed lease time is set.

Second pitfall – lock release failure

Network glitches or process crashes may prevent the lock from being released. If the lock has no lease time, the job may become permanently blocked; with a lease, the next run must wait for expiration. A lock only reduces duplicate execution – it does **not** guarantee exactly‑once semantics. The real safeguard is idempotent business logic.

Third pitfall – large tasks without sharding

Processing a million orders in a single @Scheduled method exhausts JVM memory and makes the job run for hours, while other nodes stay idle. The proper approach is to split the job into shards:

// Example shard identifiers
Shard 1: IDs 1‑10000
Shard 2: IDs 10001‑20000
Shard 3: IDs 20001‑30000
// Or hash‑based sharding
order_id % 10 = 0
order_id % 10 = 1
order_id % 10 = 2

Each shard is dispatched to a worker via a task queue, allowing parallel processing and independent retries.

Task table – persistent job metadata

A dedicated job_instance table records execution details:

id, job_type, biz_date, status, total_count,
success_count, fail_count, start_time,
finish_time, retry_count

Status values: CREATED, RUNNING, SUCCESS, PARTIAL_SUCCESS, FAILED. A unique constraint on job_type + biz_date guarantees only one record is inserted even if the scheduler fires multiple times.

Database unique constraint vs. distributed lock

For once‑a‑day business tasks, a unique key in the database combined with a try/catch for DuplicateKeyException is more reliable than a Redis lock because the database is the source of truth even if Redis restarts or the application crashes.

try {
    jobRepository.create("DAILY_SETTLEMENT", LocalDate.now());
} catch (DuplicateKeyException e) {
    return; // another node already created the job
}

Failure categories

Scheduling failure : worker not started, task not dispatched, MQ send error – should be auto‑retried.

System failure : DB timeout, Redis unavailable, third‑party timeout – may use exponential back‑off.

Business failure : invalid order state, account frozen – often non‑retryable and may need manual intervention.

An enum can encode the type:

public enum JobErrorType {
    RETRYABLE,
    NON_RETRYABLE,
    MANUAL_REQUIRED
}

When @Scheduled alone is sufficient

For tiny, short‑lived jobs that are naturally idempotent (e.g., cleaning local temp files, refreshing a local cache, collecting JVM metrics), using plain @Scheduled on each node is fine. Adding a lock is optional if occasional duplication is acceptable.

Recommended layered architecture for real‑world distributed tasks

Scheduler
  ↓
Job Instance
  ↓
Shard Generator
  ↓
Task Queue
  ↓
Worker
  ↓
Business Service
  ↓
Result Record

The Scheduler decides *when* to start; the Job Instance records the execution; the Shard Generator splits the work; the Task Queue distributes shards to Workers; Workers invoke Business Services that must be idempotent; Result Records store success/failure for audit and retry.

Conclusion

Moving from a single‑machine @Scheduled job to a clustered environment reveals the difference between a simple timer and a full‑featured distributed task system. Key questions become:

Is duplicate execution safe?

How are failures recovered?

How to split large workloads?

How to achieve parallelism after scaling?

When to upgrade to a dedicated scheduling platform (e.g., XXL‑JOB, Quartz cluster, ElasticJob) or a custom task platform?

Addressing these questions and applying the patterns above turns a fragile timer into a reliable, maintainable task platform.