How to Use XXL‑JOB in SpringBoot for Flexible Sharding of Massive Data

Scenario

An application needs to batch‑process a huge table with three requirements: parallel processing of data blocks, flexible concurrency control, and balanced load across server nodes.

Solution Idea

Because the same table must be processed in parallel, the author proposes sharding the query by taking the modulo of the primary‑key id. This prevents duplicate handling. XXL‑JOB’s routing strategies “sharding broadcast” and “dynamic sharding” match the scenario.

Implementation

1. Configure a sharding‑broadcast task in the XXL‑JOB admin console

Set the routing strategy to “sharding broadcast”.

2. Write the task code

In a SpringBoot service the task obtains the total number of shards and the current shard index from XxlJobHelper, builds an OrderDataParam, queries the corresponding slice with MyBatis, logs progress, and records execution time.

@Resource
private OrderDataMapper orderDataMapper;

@XxlJob("orderDataStatusTask")
public void orderDataStatusTask() {
    // timer
    Stopwatch timer = Stopwatch.createStarted();

    // get shard info
    OrderDataParam param = new OrderDataParam();
    param.setShardIndex(XxlJobHelper.getShardIndex());
    param.setShardTotal(XxlJobHelper.getShardTotal());

    // fetch data for this shard
    List<OrderData> orderDataList = orderDataMapper.getInitStatusOrder(this.getJobParams());
    XxlJobHelper.log("获取待处理订单数据:分片号={},数据量={},总分片数={}",
        XxlJobHelper.getShardIndex(), orderDataList.size(), XxlJobHelper.getShardTotal());
    if (CollUtil.isEmpty(orderDataList)) {
        return;
    }

    // processing logic ...

    XxlJobHelper.log("当前分片({})处理完成,耗时={}秒",
        XxlJobHelper.getShardIndex(), timer.stop().elapsed(TimeUnit.SECONDS));
}

Four service instances are started; ShardTotal = 4 and each instance receives ShardIndex 0‑3.

3. MyBatis SQL for sharding

The SELECT uses two hash operations: first mod(t.id,10) to look at the last digit, then modulo by #{shardTotal} to match the current shard.

<select id="getInitStatusOrder" parameterType="com.xxx.OrderDataParam"
        resultType="com.xxx.OrderData">
    select id,order_no,customer_code,
    from tt_order_data t
    where t.status = 0
      and t.fail_count < #{retryCount}
      and t.update_time >= #{lastUpdateTime}
      and mod(mod(t.id,10), #{shardTotal}) = #{shardIndex}
    limit 0,200
</select>

4. Scheduling logs

Each of the four instances executes the task once and processes its own slice. Example logs from instance 3 show shard index 2, 100 records processed in 1 second; instance 4 shows shard index 3, 80 records processed in 1 second.

2024-09-25 08:31:40 ... XxlJobHelper.log("获取待处理订单数据:分片号=2,数据量=100,总分片数=4")
2024-09-25 08:31:41 ... XxlJobHelper.log("当前分片(2)处理完成,耗时=1秒")
...
2024-09-25 08:31:40 ... XxlJobHelper.log("获取待处理订单数据:分片号=3,数据量=80,总分片数=4")
2024-09-25 08:31:41 ... XxlJobHelper.log("当前分片(3)处理完成,耗时=1秒")