Root Cause Analysis and Resolution of Elasticsearch BulkProcessor Deadlock in High‑Volume MQ Consumption

The system listens to product‑change MQ messages, fetches the latest product data, and uses a BulkProcessor to batch‑update fields in an Elasticsearch (ES) cluster that is sharded into 256 partitions based on a three‑level category ID.

During the 618 promotion, MQ traffic surged, causing frequent SKU name changes and category ID updates. The bulk updates were routed to the appropriate ES shard, but the massive volume of MQ messages overwhelmed the system, leading to message backlog and consumption pauses.

Thread‑stack analysis (uploaded to fastthread.io) revealed many threads blocked on the internal lock of BulkProcessor . The ES client thread elasticsearch[scheduler][T#1] held the lock while waiting, preventing business threads from acquiring it and causing a deadlock.

Key code snippets:

BulkProcessor.builder((request, bulkListener) ->
    fullDataEsClient.getClient().bulkAsync(request, RequestOptions.DEFAULT, bulkListener), listener)
    .setBulkActions(1000)
    .setBulkSize(new ByteSizeValue(5L, ByteSizeUnit.MB))
    .setConcurrentRequests(1)
    .setFlushInterval(TimeValue.timeValueSeconds(1L))
    .setBackoffPolicy(BackoffPolicy.constantBackoff(TimeValue.timeValueSeconds(1L), 5))
    .build();

The builder creates a single‑thread ScheduledThreadPoolExecutor for both the periodic flush task and the retry scheduler:

static ScheduledThreadPoolExecutor initScheduler(Settings settings) {
    ScheduledThreadPoolExecutor scheduler = new ScheduledThreadPoolExecutor(1,
        EsExecutors.daemonThreadFactory(settings, "scheduler"), new EsAbortPolicy());
    scheduler.setExecuteExistingDelayedTasksAfterShutdownPolicy(false);
    scheduler.setContinueExistingPeriodicTasksAfterShutdownPolicy(false);
    scheduler.setRemoveOnCancelPolicy(true);
    return scheduler;
}

The flush task runs at a fixed interval (1 s) and acquires the same lock as the add method invoked by MQ consumer threads, creating lock contention. The retry logic also uses the same scheduler thread, and because the BulkProcessor is configured with concurrentRequests = 1 , only one thread can execute at a time, forming a circular wait between the flush task, the retry task, and the consumer threads.

Solution approaches:

Upgrade the Elasticsearch client to version 7.6 or later, where the retry thread pool is isolated from the flush thread pool, eliminating the contention.

Disable the retry mechanism if upgrading is not feasible, ensuring that failed bulk requests are handled by alternative business‑level retry strategies.

Both fixes break the deadlock and restore normal MQ consumption rates.