Mastering High‑Throughput Thread Pools: Strategies for 100k QPS in Java

Background

During a large‑scale promotion we encountered a workload with the following characteristics: QPS reaches 100,000, each task takes about 100 ms, and every request must be processed in a thread pool (e.g., sending notifications, logging, async persistence). The system must remain stable, respond quickly, and achieve high resource utilization.

Problem Analysis

If we use the default Java thread pool configuration such as Executors.newFixedThreadPool(100), handling 100,000 requests per second with 100 ms per task would require at least 10,000 threads (since each thread can process only 10 requests per second). This would quickly exhaust CPU and memory, leading to JVM crashes and potential system avalanche.

Optimization Goals

Control the number of threads to avoid explosion.

Increase throughput so a single thread can handle more tasks.

Use queues and rate‑limiting to prevent cascading failures.

Provide graceful degradation mechanisms such as reject policies and fallback handling.

Solution: Thread‑Pool Optimization Strategies

Strategy

Description

Task batch processing

Merge tasks arriving within 100 ms into batches for collective execution, improving efficiency.

Proper thread‑pool parameters

Configure core pool size, maximum pool size, and queue capacity according to workload.

Async + rate limiting

Limit the rate of task submission to protect the pool from being overwhelmed.

Custom rejection policy

Log rejected tasks, trigger alerts, and optionally enqueue them for later processing.

Implementation: Spring Boot Custom Thread Pool

@Configuration
public class ThreadPoolConfig {
    @Bean("taskExecutor")
    public ThreadPoolExecutor taskExecutor() {
        int corePoolSize = 200;      // core threads
        int maxPoolSize = 500;       // max threads
        int queueCapacity = 10000;  // queue size
        int keepAliveTime = 60;     // seconds for non‑core threads
        return new ThreadPoolExecutor(
                corePoolSize,
                maxPoolSize,
                keepAliveTime,
                TimeUnit.SECONDS,
                new LinkedBlockingQueue<>(queueCapacity),
                r -> new Thread(r, "task-executor-" + new AtomicInteger().getAndIncrement()),
                (r, executor) -> {
                    System.err.println("Task rejected: " + r.toString());
                    // additional alert or fallback logic
                }
        );
    }
}

Business Layer: Asynchronous Task Submission

@Service
public class TaskService {
    @Resource(name = "taskExecutor")
    private ThreadPoolExecutor taskExecutor;

    /**
     * Receive high‑concurrency requests and process tasks asynchronously.
     * @param taskId Task identifier
     */
    public void submitTask(String taskId) {
        taskExecutor.execute(() -> {
            long start = System.currentTimeMillis();
            try {
                // Simulate task processing (e.g., remote call, DB operation)
                Thread.sleep(100);
                System.out.println(Thread.currentThread().getName() +
                        " executed task [" + taskId + "] successfully, cost: " +
                        (System.currentTimeMillis() - start) + "ms");
            } catch (InterruptedException e) {
                Thread.currentThread().interrupt();
            }
        });
    }
}

Controller: Simulated Load Test Endpoint

@RestController
@RequestMapping("/api/task")
public class TaskController {
    @Autowired
    private TaskService taskService;

    /**
     * Simulate high‑concurrency task submission.
     * @param taskId Task identifier
     */
    @PostMapping("/submit")
    public ResponseEntity<String> submit(@RequestParam String taskId) {
        taskService.submitTask(taskId);
        return ResponseEntity.ok("Task submitted");
    }
}

Advanced Optimization Suggestions

Disruptor or LMAX RingBuffer : For millisecond‑level, ultra‑high‑throughput scenarios, a single‑thread lock‑free queue can achieve millions of QPS.

Asynchronous batch processing : Collect tasks every 100 ms and process them together, reducing context switches and improving CPU utilization.

Rate limiting + circuit breaking + fallback : Use Sentinel or Resilience4j to limit the entry rate (e.g., 100k requests/s), apply circuit‑breaker rules on consecutive failures, and provide graceful degradation.

Conclusion

In a 100k QPS, 100 ms task scenario, thread‑pool optimization is essential. By analyzing business characteristics, tuning core pool size, maximum pool size, queue capacity, and implementing batch processing, custom rejection handling, and advanced async mechanisms, we can control concurrency, boost per‑thread efficiency, and ensure system stability under extreme load.