Why @Scheduled Runs 3× in Kubernetes: Migrating to JobRunr for Distributed Tasks

Problem: @Scheduled Runs Multiple Times in Kubernetes

A Spring Boot application originally ran as a single instance. After migrating to Kubernetes and setting replicas: 3 for high availability, a daily report generation task annotated with @Scheduled(cron = "0 0 2 * * ?") executed three times simultaneously — once per pod — producing three identical reports. The root cause: @Scheduled uses a JVM-internal scheduler unaware of other identical Spring Boot instances.

Initial Fix Considered: Distributed Locks

The common solution is a distributed lock (e.g., Redis SETNX, database lock, or ShedLock). For a few simple cron jobs this suffices. However, the project also had:

Order timeout closure: @Scheduled(fixedDelay = 60_000) Inventory sync: @Scheduled(cron = "0 */5 * * * ?") Async Excel exports triggered by user clicks: @Async public void export(Long exportId) These requirements expanded beyond "prevent duplicate execution" to include: persistence across JVM restarts, automatic retry with backoff, delayed execution (e.g., 30 minutes later), task status visibility, manual re-execution, and knowing which instance processed a task. Building all that manually would essentially mean writing a custom task table and scheduler.

Choosing JobRunr

JobRunr was selected because:

No separate scheduling server needed; uses existing MySQL/PostgreSQL/MongoDB for storage.

Spring Boot 4 starter available ( jobrunr-spring-boot-4-starter:8.8.1).

Recommended in Spring's official August newsletter.

Maven dependency:

org.jobrunr
  jobrunr-spring-boot-4-starter
  8.8.1

Configuration (YAML):

jobrunr:
  background-job-server:
    enabled: true
    worker-count: 8
  dashboard:
    enabled: true
    port: 8000
  jobs:
    default-number-of-retries: 5
    delete-succeeded-jobs-after: 36h
    permanently-delete-deleted-jobs-after: 72h

The starter reuses the application's DataSource. On first startup JobRunr creates its own tables. The default retry count is 10; here reduced to 5.

Migrating Fire-and-Forget Jobs (Excel Export)

Original async method:

@Async
public void export(Long exportId) { ... }

Problem: task lives only in JVM memory; restart loses it.

New approach using JobScheduler (preferred over static BackgroundJob for testability):

@Service
public class ExportApplicationService {
  private final ExportService exportService;
  private final JobScheduler jobScheduler;
  private final ExportJob exportJob;
  
  public ExportApplicationService(ExportService exportService,
                                  JobScheduler jobScheduler,
                                  ExportJob exportJob) {
    this.exportService = exportService;
    this.jobScheduler = jobScheduler;
    this.exportJob = exportJob;
  }
  
  public Long submit(ExportRequest request) {
    Long exportId = exportService.create(request);
    jobScheduler.enqueue(() -> exportJob.execute(exportId));
    return exportId;
  }
}

Job implementation:

@Component
public class ExportJob {
  private final ExportService exportService;
  
  public ExportJob(ExportService exportService) {
    this.exportService = exportService;
  }
  
  @Job(name = "生成订单导出文件")
  public void execute(Long exportId) {
    exportService.generate(exportId);
  }
}

Key practice: pass only the business ID ( exportId), not large DTOs or entities. This keeps serialized job parameters small, stable, and compatible with future schema changes. The job fetches fresh data at execution time.

Failure handling: when remoteFileService.upload() throws an exception, JobRunr automatically retries with exponential backoff (default). After exhausting retries (5 here), the job enters FAILED state and remains visible in the dashboard with full stack trace.

Dashboard Monitoring

Access http://localhost:8000 to see task states: Enqueued, Processing, Succeeded, Failed. Failed tasks show exception and stack trace. This replaces the previous manual hunt across logs and pods. Note: dashboard has no authentication by default; secure it in production via gateway, network policies, or JobRunr's built-in username/password.

Migrating Recurring Jobs (Cron / Fixed Delay)

Old order timeout scanner:

@Scheduled(fixedDelay = 60_000)
public void closeExpiredOrders() {
  orderService.closeExpiredOrders();
}

New recurring job:

@Component
public class OrderMaintenanceJob {
  private final OrderService orderService;
  
  public OrderMaintenanceJob(OrderService orderService) {
    this.orderService = orderService;
  }
  
  @Recurring(id = "close-expired-orders", cron = "*/5 * * * *")
  @Job(name = "关闭超时订单")
  public void closeExpiredOrders() {
    orderService.closeExpiredOrders();
  }
}

The recurring definition is stored in the shared database. The cluster elects one node to schedule the recurring job; workers execute it. JobRunr OSS avoids concurrent overlapping executions of the same recurring job by default.

Delayed Jobs (Schedule for Future Execution)

Example: send real-name reminder 30 minutes after user registration if not completed.

Old way: insert row with execute_time, then a polling @Scheduled scans every minute.

JobRunr way:

jobScheduler.schedule(
  Instant.now().plus(Duration.ofMinutes(30)),
  () -> reminderJob.sendRealNameReminder(userId)
);

The job is persisted with a future timestamp and executed automatically. JobRunr supports fire-and-forget, delayed, and recurring jobs uniformly.

This allowed deleting a dedicated @Scheduled that polled a WAITING task table. The code now directly expresses the business requirement: "execute 30 minutes later."

When to Use JobRunr vs Simple Locks

If the project only has 2-3 simple cron jobs that run for seconds, @Scheduled + ShedLock is perfectly adequate. JobRunr shines when tasks require persistence, retries, delayed execution, cluster-wide deduplication, status tracking, and manual intervention.

Production Hardening: Idempotency & Deduplication

Business Idempotency

JobRunr guarantees at-least-once execution. If a job succeeds but the framework crashes before marking it complete, the job may run again. Business methods with side effects must be idempotent. Example: coupon issuance table with unique constraint:

UNIQUE KEY uk_user_activity(user_id, activity_id)

Applies to: coupon issuance, billing, SMS, payment calls, settlement records. Scheduler reliability does not replace business-level idempotency.

Duplicate Submission Prevention

User double-clicks "Generate Report" → two enqueue calls. JobRunr OSS supports deterministic job IDs via UUID:

UUID jobId = UUID.nameUUIDFromBytes(("export:" + exportId).getBytes(StandardCharsets.UTF_8));
jobScheduler.enqueue(jobId, () -> exportJob.execute(exportId));

Same exportId yields same jobId, preventing duplicate queueing.

Conclusion

@Scheduled

itself is not flawed; it's designed for single-JVM scheduling. As the system scales to multiple pods and task complexity grows, tasks become data that needs management: creation time, execution time, retry count, assigned worker, survival across restarts. JobRunr provides that management layer without heavy infrastructure. The migration was incremental: local maintenance tasks (e.g., cache refresh) stay on @Scheduled; cluster-wide business tasks move to JobRunr. Now scaling replicas: 3 to replicas: 6 no longer requires auditing every @Scheduled annotation.