Boost High-Concurrency Performance with Request Merging in Java

Introduction

In many projects we call third‑party APIs and also expose our own APIs (RPC, HTTP, etc.). Providing batch APIs can improve performance, especially under high concurrency.

Without Merged Requests

Before merging, each request is handled individually, which is simple but inefficient for high‑frequency endpoints such as a movie‑detail service.

Example: device registration interface.

@Reference(check = false)
private DeviceService deviceService;

/**
 * Register device
 *
 * @param productKey product key
 * @param deviceName device name
 * @return device ID
 */
public R<Long> registDevice(String productKey, String deviceName) {
    log.debug("Start registration: {}, {}", productKey, deviceName);
    DeviceRequestDto deviceCreateQuery = new DeviceRequestDto()
                    .setProductKey(productKey)
                    .setName(deviceName);
    Long deviceId = deviceService.createDevice(deviceCreateQuery);
    return deviceId != null ? R.ok(deviceId) : R.error(DEVICE_CREATE_ERROR);
}

Request Merging

Merging requests can boost performance but adds complexity, latency, and risk; it should be applied only after careful business analysis.

The idea: store incoming requests, periodically invoke a batch interface, then notify each original request.

Implementation

1. Batch Interface

First we need a batch‑creation API on the service side.

Method signature

/**
 * Batch create device interface
 *
 * @param deviceRequestDtoList input list
 * @return creation result
 */
R<List<DeviceCreateResp>> batchCreateDevice(List<DeviceCreateQuery> deviceList);

Parameter class

@Data
public class DeviceCreateQuery implements Serializable {
    /** product identifier */
    private String productKey;
    /** device name */
    private String name;
    /** request source, unique per batch */
    private String requestSource;
}

Response class

@Data
public class DeviceCreateResp implements Serializable {
    /** device ID */
    private Long deviceId;
    /** request source, unique per batch */
    private String requestSource;
}

2. Merging Single Requests

Blocking queue

private LinkedBlockingDeque<DeviceCreateRequest> deviceCreateQueue = new LinkedBlockingDeque<>();

Custom request structure

@Data
static class DeviceCreateRequest {
    /** product key */
    private String productKey;
    /** device name */
    private String deviceName;
    /** request source, must be unique */
    private String requestSource;
    /** CompletableFuture for the result */
    private CompletableFuture<Long> completedFuture;
}

Storing a request

public R<Long> registDevice(String productKey, String deviceName) {
    log.debug("Start registration: {}, {}", productKey, deviceName);
    // cache request ====== start
    CompletableFuture<Long> completedFuture = new CompletableFuture<>();
    DeviceCreateRequest deviceCreateRequest = new DeviceCreateRequest();
    deviceCreateRequest.setProductKey(productKey);
    deviceCreateRequest.setDeviceName(deviceName);
    deviceCreateRequest.setRequestSource(UUID.randomUUID().toString());
    deviceCreateRequest.setCompletedFuture(completedFuture);
    deviceCreateQueue.add(deviceCreateRequest);
    // cache request ====== end
    Long deviceId = null;
    try {
        deviceId = completedFuture.get();
    } catch (Exception e) {
        log.error("Device registration failed", e);
    }
    return deviceId != null ? R.ok(deviceId) : R.error(DEVICE_CREATE_ERROR);
}

3. Sending Batch Requests

A scheduled thread pool periodically drains the queue, builds a list of DeviceCreateQuery, calls the batch API, and completes the corresponding futures.

/** Blocking queue for cached requests */
private LinkedBlockingDeque<DeviceCreateRequest> deviceCreateQueue = new LinkedBlockingDeque<>();
/** Number of threads in the pool */
@Value("${iot.register.merge.device.request.num:100}")
private int createDeviceMergeNum;
/** Period between executions (ms) */
@Value("${iot.register.merge.device.request.period:30}")
private long createDeviceMergePeriod;

@Reference(check = false)
private DeviceService deviceService;

@PostConstruct
public void init() {
    ScheduledExecutorService scheduledExecutorService = Executors.newScheduledThreadPool(createDeviceMergeNum);
    scheduledExecutorService.scheduleAtFixedRate(() -> {
        // snapshot of the queue
        List<DeviceCreateRequest> questBak = new ArrayList<>();
        List<DeviceCreateQuery> deviceCreateQueryList = new ArrayList<>();
        int size = deviceCreateQueue.size();
        for (int i = 0; i < size; i++) {
            DeviceCreateRequest deviceCreateRequest = deviceCreateQueue.poll();
            if (Objects.nonNull(deviceCreateRequest)) {
                questBak.add(deviceCreateRequest);
                deviceCreateQueryList.add(buildDeviceCreateQuery(deviceCreateRequest));
            }
        }
        if (!deviceCreateQueryList.isEmpty()) {
            try {
                List<DeviceCreateResp> response = deviceService.batchCreateDevice(deviceCreateQueryList);
                Map<String, Long> collect = response.stream()
                        .collect(Collectors.toMap(DeviceCreateResp::getRequestSource, DeviceCreateResp::getDeviceId));
                // notify original threads
                for (DeviceCreateRequest deviceCreateRequest : questBak) {
                    deviceCreateRequest.getCompletedFuture().complete(collect.get(deviceCreateRequest.getRequestSource()));
                }
            } catch (Throwable throwable) {
                log.error("Batch device registration error", throwable);
                // notify with exception
                questBak.forEach(req -> req.getCompletedFuture().obtrudeException(throwable));
            }
        }
    }, 0, createDeviceMergePeriod, TimeUnit.MILLISECONDS);
}

Conclusion

Request merging is a practical technique for high‑concurrency scenarios; this article presents a simple Java implementation using BlockingDeque and CompletableFuture. Production systems often rely on mature frameworks such as Hystrix for more robust merging.