Boost Java Batch Updates with Multithreading: A Practical Guide

Introduction

Many developers face performance bottlenecks when performing batch operations on massive data sets. Using multithreading to execute these tasks can dramatically improve efficiency, and the implementation is surprisingly straightforward.

Overall Flow Diagram

The diagram shows the simple processing flow: split the large collection, execute batch updates in parallel, and control the execution order.

Steps

Obtain the large collection A that needs batch updating and split it into N smaller collections A‑1 … A‑N.

Start a thread pool, tune its parameters according to the size of the collections, and perform batch updates on each small collection.

Control the execution order of the threads to ensure proper workflow.

Utility Class for Splitting Lists

import com.google.common.collect.Lists;
import org.apache.commons.collections.CollectionUtils;
import java.util.List;

/**
 * Utility for splitting collections.
 */
public class SplitListUtils {
    /**
     * Split a collection into sub‑lists.
     * @param <T> generic type
     * @param resList the collection to split
     * @param subListLength size of each sub‑list
     * @return a list containing the split sub‑lists
     */
    public static <T> List<List<T>> split(List<T> resList, int subListLength) {
        if (CollectionUtils.isEmpty(resList) || subListLength <= 0) {
            return Lists.newArrayList();
        }
        List<List<T>> ret = Lists.newArrayList();
        int size = resList.size();
        if (size <= subListLength) {
            // Not enough data to split
            ret.add(resList);
        } else {
            int pre = size / subListLength;
            int last = size % subListLength;
            // First 'pre' sub‑lists each have 'subListLength' elements
            for (int i = 0; i < pre; i++) {
                List<T> itemList = Lists.newArrayList();
                for (int j = 0; j < subListLength; j++) {
                    itemList.add(resList.get(i * subListLength + j));
                }
                ret.add(itemList);
            }
            // Handle the remaining elements
            if (last > 0) {
                List<T> itemList = Lists.newArrayList();
                for (int i = 0; i < last; i++) {
                    itemList.add(resList.get(pre * subListLength + i));
                }
                ret.add(itemList);
            }
        }
        return ret;
    }

    // Example usage
    public static void main(String[] args) {
        List<String> list = Lists.newArrayList();
        int size = 1099;
        for (int i = 0; i < size; i++) {
            list.add("hello-" + i);
        }
        List<List<String>> temps = split(list, 100);
        int j = 0;
        for (List<String> obj : temps) {
            System.out.println(String.format("row:%s -> size:%s,data:%s", ++j, obj.size(), obj));
        }
    }
}

Asynchronous Task Execution with Thread Pool

public void threadMethod() {
    List<T> updateList = new ArrayList<>();
    // Initialize thread pool – parameters must be tuned carefully!
    ThreadPoolExecutor threadPool = new ThreadPoolExecutor(
            20, 50, 4, TimeUnit.SECONDS,
            new ArrayBlockingQueue<>(10), new ThreadPoolExecutor.AbortPolicy());

    // Split the large collection into N small collections (e.g., size 100)
    List<T> splitNList = SplitListUtils.split(totalList, 100);
    CountDownLatch countDownLatch = new CountDownLatch(splitNList.size());

    // Process each sub‑list in parallel
    for (List<T> singleList : splitNList) {
        threadPool.execute(new Thread(new Runnable() {
            @Override
            public void run() {
                for (Entity yangshiwen : singleList) {
                    // Package each object into an update list
                    // ...
                }
            }
        }));
        countDownLatch.countDown();
    }
    try {
        // Wait until all tasks finish
        countDownLatch.await();
    } catch (InterruptedException e) {
        throw new BusinessLogException(ResponseEnum.FAIL);
    }
    // Batch insert via MyBatis (ensure the list is not empty)
    if (GeneralUtil.listNotNull(updateList)) {
        batchUpdateEntity(updateList);
        LogUtil.info("Batch update completed");
    }
}

Conclusion

Multithreading in Java can be challenging but also rewarding; remember to shut down your thread pool after use.