Tired of XXL‑Job? Try This Elegant Nacos‑Based Scheduling Solution

Preface

XXL‑Job is a popular Java task scheduler, but when a system already adopts Nacos for service discovery and configuration, keeping XXL‑Job’s own registry and config creates friction.

Challenges in the Nacos Ecosystem

Challenge 1 – Dual Registries Cause State Inconsistency

Each executor reports its status to both Nacos and XXL‑Job. If an instance is taken offline in Nacos, XXL‑Job still thinks it is online, leading to situations such as:

you: click "offline" in Nacos → Nacos: instance offline ✓
you: start JVM dump
XXL‑Job: still schedules tasks to this instance

Similar mismatches occur with network flaps, restarts, or gray releases.

Challenge 2 – Lack of Observability

Debugging a failed job requires checking the Admin UI logs, the executor’s logs, and hoping the timestamps line up. Without a unified TraceId, the investigation is guesswork.

Challenge 3 – Weak Sharding Guarantees

XXL‑Job’s sharding is advisory; two executors may process the same data because there is no distributed lock protecting the shard range.

int shardIndex = XxlJobHelper.getShardIndex(); // 0
int shardTotal = XxlJobHelper.getShardTotal(); // 10
List<Order> orders = orderDao.findByIdMod(shardIndex, shardTotal);

In production, an executor restart can cause duplicate processing.

Core Idea – Middleware as Business

Instead of treating middleware as a separate platform, embed scheduling capabilities directly into the microservice ecosystem.

From "Heavy Middleware" to "Light Capability"

Traditional architecture:

Business services → call → XXL‑Job Admin (stand‑alone deployment)

In cloud‑native environments this adds deployment, monitoring, and configuration overhead.

JobFlow’s philosophy:

Business services → call → JobFlow Scheduler (microservice)

All services share the same container, K8s, Prometheus, Grafana, Nacos Config, and logging stack.

Reduce Redundancy (Subtract)

Remove XXL‑Job’s built‑in registry; use Nacos service discovery exclusively.

Store only task definitions, execution records, and audit logs in MySQL; do not store service registration or scheduler config there.

Add Missing Capabilities (Add)

Full‑link TraceId injected into HTTP headers and propagated to the executor’s MDC.

True sharding with explicit data ranges protected by a Redis distributed lock.

Intelligent retry with exponential back‑off and dead‑letter queue.

Scheduler configuration lives in Nacos Config, supporting dynamic updates, multi‑instance sharing, and version rollback.

Out‑of‑the‑box Prometheus metrics, RESTful API for manual trigger, query, and retry.

JobFlow Architecture

Overall Structure

Nacos : unified service discovery & configuration.

JobFlow Scheduler : lightweight microservice scheduler.

MySQL : task definitions, execution history, audit logs.

The scheduler runs as a regular microservice, automatically reusing Actuator, Prometheus, alerting, and log collection.

Call Flow

Scheduler generates a UUID traceId and sets HTTP headers X-Trace-Id, X-Shard-Index, X-Shard-Total.

Executor receives the headers, stores traceId in MDC, logs with the traceId.

All logs can be searched by traceId in ELK/Loki, showing trigger time, target executor, processing details, and errors.

This end‑to‑end trace reduces troubleshooting time by an order of magnitude.

Sharding Scheduling

int totalRecords = 1_000_000;
int shardTotal = 10;
int rangeSize = totalRecords / shardTotal;
for (int i = 0; i < shardTotal; i++) {
    long startId = i * rangeSize;
    long endId = (i + 1) * rangeSize - 1;
    String lockKey = String.format("lock:job:order-sync:range:%d-%d", startId, endId);
    // build request with traceId, startId, endId, lockKey
    executeAsync(instance, request);
}

Executor side:

@PostMapping("/internal/job/order-sync")
public JobResult sync(@RequestHeader("X-Start-Id") Long startId,
                     @RequestHeader("X-End-Id") Long endId,
                     @RequestHeader("X-Lock-Key") String lockKey) {
    boolean locked = redisLock.tryLock(lockKey, 60, TimeUnit.SECONDS);
    if (!locked) {
        log.warn("Shard {}-{} already locked", startId, endId);
        return JobResult.skip("already processed");
    }
    try {
        List<Order> orders = orderDao.findByIdBetween(startId, endId);
        // business logic
        return JobResult.success();
    } finally {
        redisLock.unlock(lockKey);
    }
}

The lock guarantees that each data range is processed by only one instance, even after restarts.

Key Features Detailed

Feature 1 – Full‑Link TraceId

String traceId = UUID.randomUUID().toString();
HttpHeaders headers = new HttpHeaders();
headers.set("X-Trace-Id", traceId);
// send request

Executor stores it in MDC, making every log line contain the traceId. Searching the traceId in ELK yields the complete execution chain.

Feature 2 – True Sharding

Explicit range calculation + Redis lock ensures no overlap and supports resumable processing.

Feature 3 – Intelligent Retry

# retry config (application.yml)
retry:
  max: 5
  backoff: EXPONENTIAL
  initialDelay: 1s
  maxDelay: 5m

The scheduler computes exponential delay; after max retries the job is sent to a dead‑letter queue.

Feature 4 – Cloud‑Native Scheduler Configuration

# jobflow-scheduler.yaml (Nacos Config)
jobflow:
  scheduler:
    thread-pool-size: 20
    timeout: 300
    max-retry: 3
  executor:
    connect-timeout: 5000
    read-timeout: 30000
  redis:
    lock-timeout: 60
  compensation:
    enabled: true
    interval: 60000
    stuck-threshold: 600000

Changing thread-pool-size in the Nacos console instantly updates all scheduler instances without restart; version rollback is also one‑click.

Feature 5 – Simplified Database Design

CREATE TABLE job_definition (
  id BIGINT PRIMARY KEY AUTO_INCREMENT,
  job_name VARCHAR(100) UNIQUE,
  service_name VARCHAR(100),
  handler VARCHAR(100),
  cron VARCHAR(100),
  enabled BOOLEAN DEFAULT TRUE,
  created_at TIMESTAMP,
  updated_at TIMESTAMP
);

CREATE TABLE job_execution (
  id BIGINT PRIMARY KEY AUTO_INCREMENT,
  job_name VARCHAR(100) NOT NULL,
  trace_id VARCHAR(64) NOT NULL UNIQUE,
  trigger_time TIMESTAMP NOT NULL,
  finish_time TIMESTAMP,
  status VARCHAR(20) NOT NULL,
  retry_count INT DEFAULT 0,
  result_message TEXT,
  INDEX idx_trace (trace_id),
  INDEX idx_job_time (job_name, trigger_time)
);

The DB stores only definitions, execution metadata, and audit logs; service registration and scheduler config remain in Nacos, keeping the DB lightweight and fast.

Common Q&A

Q1: What if Nacos goes down?

Use a local cache of service instances with Guava’s LoadingCache. When Nacos throws an exception, fall back to the cached list so scheduling can continue temporarily.

@Service
public class ExecutorDiscovery {
    private LoadingCache<String, List<String>> cache = CacheBuilder.newBuilder()
        .expireAfterWrite(5, TimeUnit.MINUTES)
        .build(key -> namingService.getAllInstances(key));

    public List<String> getInstances(String serviceName) {
        try {
            return namingService.getAllInstances(serviceName);
        } catch (NacosException e) {
            log.warn("Nacos unavailable, using cache");
            return cache.getIfPresent(serviceName);
        }
    }
}

Q2: How to handle DB write failures that cause state mismatch?

Adopt eventual consistency: write a PENDING record first, invoke the executor asynchronously, then update to SUCCESS/FAILED. A background compensation task scans stale PENDING rows and reconciles them using the traceId.

// Insert PENDING
jobExecutionDao.insert(new JobExecution().setTraceId(traceId)
    .setStatus("PENDING").setTriggerTime(now));
// Async execution
CompletableFuture.runAsync(() -> {
    try {
        JobResult r = executeJob(...);
        jobExecutionDao.updateStatus(traceId, r.getStatus());
    } catch (Exception e) {
        jobExecutionDao.updateStatus(traceId, "FAILED");
    }
});

// Compensation task (every minute)
@Scheduled(fixedDelay = 60000)
public void fixStuckExecutions() {
    List<JobExecution> stuck = jobExecutionDao.findStuckExecutions();
    // inspect logs via traceId or mark TIMEOUT
}

Q3: No UI for operations?

Expose RESTful endpoints for manual trigger, history query, detail lookup by traceId, and retry. Swagger UI can be added later.

@RestController
@RequestMapping("/api/jobs")
public class JobController {
    @PostMapping("/{name}/trigger")
    public JobResult trigger(@PathVariable String name) { return jobService.triggerNow(name); }

    @GetMapping("/{name}/executions")
    public Page<JobExecution> history(@PathVariable String name,
                                      @RequestParam int page,
                                      @RequestParam int size) {
        return jobExecutionDao.findByJobName(name, PageRequest.of(page, size));
    }

    @GetMapping("/executions/{traceId}")
    public JobExecution detail(@PathVariable String traceId) { return jobExecutionDao.findByTraceId(traceId); }

    @PostMapping("/executions/{traceId}/retry")
    public JobResult retry(@PathVariable String traceId) { return jobService.retry(traceId); }
}

Q4: How to guarantee high availability of the scheduler?

Make the scheduler stateless and run multiple instances. Use a distributed lock per job to avoid duplicate triggers.

@Scheduled(cron = "${job.cron}")
public void scheduledTrigger() {
    List<JobConfig> jobs = getEnabledJobs();
    for (JobConfig job : jobs) {
        String lockKey = "lock:schedule:" + job.getName();
        boolean locked = redisLock.tryLock(lockKey, 10, TimeUnit.SECONDS);
        if (locked) {
            try { trigger(job); } finally { redisLock.unlock(lockKey); }
        }
    }
}

Conclusion

JobFlow is a conceptual prototype that demonstrates the "middleware as business" mindset. It shows how, in a Nacos‑centric cloud‑native stack, a scheduler can be stripped of redundant components, gain full observability via TraceId, enforce strict sharding with distributed locks, provide intelligent retry, and be configured dynamically through Nacos Config. While XXL‑Job remains a robust general‑purpose scheduler, JobFlow offers a tighter integration for teams that have already deep‑wired Nacos into their microservice ecosystem.