Thread‑Pool Outage Postmortem: Four Defense Layers to Prevent Data Loss

On July 13, a sudden surge in order‑status‑sync traffic filled a shared thread‑pool queue (capacity 1000) used by three methods in the thirdparty service, causing the pool’s worker threads to become busy and subsequent refund‑order‑sync and order‑sync tasks to be rejected, leading to rising response times and delayed downstream data.

First defense – Thread‑pool isolation : the original single pool was replaced with three independent pools, each configured with its own core and max thread counts and a 1000‑capacity queue. Example code:

private final ThreadPoolUtils orderSyncPool = new ThreadPoolUtils(5, 20, 1000, "订单同步线程池");
private final ThreadPoolUtils refundOrderSyncPool = new ThreadPoolUtils(3, 10, 1000, "退款单同步线程池");
private final ThreadPoolUtils orderStatusSyncPool = new ThreadPoolUtils(5, 15, 1000, "订单状态同步线程池");

This isolation ensures that a traffic spike in a non‑core business (order‑status sync) cannot drag down core services (order sync).

Second defense – CallerRunsPolicy with structured alerts : the previous custom rejection handler only logged a warning and silently dropped tasks. The new handler extends ThreadPoolExecutor.CallerRunsPolicy, logs an ERROR with structured fields (pool name, active threads, queue size, trigger count), and then executes the task in the submitting thread.

// Original reject handler – only logs, discards task
public void rejectedExecution(Runnable r, ThreadPoolExecutor e) {
    log.warn("线程池队列已满，任务被拒绝");
    // no super.rejectedExecution, task dropped
}

// Custom CallerRunsPolicy
public class CustomCallerRunsPolicy extends ThreadPoolExecutor.CallerRunsPolicy {
    private final AtomicInteger callerRunsCount = new AtomicInteger(0);
    @Override
    public void rejectedExecution(Runnable r, ThreadPoolExecutor e) {
        int count = callerRunsCount.incrementAndGet();
        StructuredLog.error(log)
            .moduleCode("THREAD_POOL")
            .moduleName(poolName)
            .eventCode("POOL_FULL")
            .eventName("线程池队列已满，由调用线程同步执行")
            .put("activeThreads", e.getActiveCount())
            .put("queueSize", e.getQueue().size())
            .put("callerRunsCount", count)
            .status("error")
            .log();
        super.rejectedExecution(r, e);
    }
}

When the queue is full, the submitting HTTP or Dubbo thread runs the task, preventing silent data loss at the cost of increased response latency, which acts as natural back‑pressure.

Third defense – Minute‑level DingTalk alert group : a dedicated DingTalk group receives alerts generated by the structured logs. The alert pipeline is: queue full → custom policy logs ERROR → log platform matches POOL_FULL event → DingTalk bot pushes a message. Alerts contain pool name, active thread count, queue size, and trigger count, reaching developers within one minute.

Fourth defense – Data‑compensation tool : because ArrayBlockingQueue is in‑memory and lost on JVM restart, a compensation utility reprocesses missed records. An interceptor persists every request’s full JSON payload into the third_docs_record table before business logic runs:

// Interceptor pre‑persistence
if (flag) {
    ThirdDocsRecordDTO record = ThirdDocsRecordDTO.builder()
        .businessNo(extractBusinessNo(params))
        .servletPath(servletPath)
        .param(requestBody) // full request JSON
        .nonce(nonce)
        .build();
    thirdDocsRecordService.save(record);
    return true;
}

The compensation tool queries records by time range and document numbers, checks whether downstream systems have already received each record, and re‑invokes the sync method for those that haven’t:

public void compensate(Date startTime, Date endTime, List<String> docsNoList) {
    List<ThirdDocsRecordPO> records = recordMapper.selectByTimeRangeAndDocsNos(startTime, endTime, docsNoList);
    for (ThirdDocsRecordPO record : records) {
        if (isAlreadySynced(record.getBusinessNo())) {
            continue; // already processed
        }
        replaySync(record.getParam()); // push again
    }
}

Idempotency is guaranteed by downstream deduplication logic, so re‑pushed requests do not create duplicates. Even if a server restart clears the in‑memory queue, the tool can restore lost data on the same day.

Summary of the four defenses :

Thread‑pool isolation – prevents non‑core traffic from affecting core services.

CallerRunsPolicy – ensures tasks are not dropped when the queue is full.

Minute‑level DingTalk alerts – developers know about failures within a minute.

Data‑compensation tool – recovers lost data after extreme failures.

After the refactor, operational experience improved: failures are detected instantly, and any data loss can be compensated the same day.