How to Optimize Java Thread Pools for 100k QPS: Strategies & Code

Background

During a large promotion activity we faced a scenario where the interface throughput reached 100,000 QPS, each task took about 100 ms, and every request needed to be processed in a thread pool (e.g., sending notifications, logging, async persistence). The system required high stability, fast response, and efficient resource utilization.

If not optimized, the thread pool would quickly exhaust CPU and memory, potentially causing a system avalanche.

Problem Analysis

Using the default Java fixed thread pool (Executors.newFixedThreadPool(100)) for 100 k QPS with 100 ms tasks would require at least 10,000 threads, leading to massive context‑switch overhead and possible JVM crash.

Optimization Goals

Control the number of threads to avoid explosion.

Increase throughput per thread.

Use queues and rate limiting to prevent avalanche.

Provide graceful degradation: rejection policies, fallback handling.

Solution: Thread‑Pool Optimization Strategies

Task batching : Merge tasks within 100 ms windows and execute them together.

Proper thread‑pool parameters : Tune core size, max size, queue capacity, keep‑alive time.

Asynchronous + rate limiting : Limit the rate of task submission to protect the pool.

Custom rejection policy : Log, downgrade, or push to a standby queue when tasks are rejected.

Code Implementation

Spring Boot custom thread‑pool configuration:

@Configuration
public class ThreadPoolConfig {

    @Bean("taskExecutor")
    public ThreadPoolExecutor taskExecutor() {
        int corePoolSize = 200;      // core threads
        int maxPoolSize = 500;       // max threads
        int queueCapacity = 10000;   // queue size
        int keepAliveTime = 60;      // seconds

        return new ThreadPoolExecutor(
                corePoolSize,
                maxPoolSize,
                keepAliveTime,
                TimeUnit.SECONDS,
                new LinkedBlockingQueue<>(queueCapacity),
                new ThreadFactory() {
                    private final AtomicInteger count = new AtomicInteger(0);
                    @Override
                    public Thread newThread(Runnable r) {
                        return new Thread(r, "task-executor-" + count.getAndIncrement());
                    }
                },
                new RejectedExecutionHandler() {
                    @Override
                    public void rejectedExecution(Runnable r, ThreadPoolExecutor executor) {
                        System.err.println("Task rejected: " + r);
                        // additional alert or fallback logic
                    }
                });
    }
}

Business layer asynchronous task submission:

@Service
public class TaskService {

    @Resource(name = "taskExecutor")
    private ThreadPoolExecutor taskExecutor;

    /** Receive high‑concurrency request and process asynchronously */
    public void submitTask(String taskId) {
        taskExecutor.execute(() -> {
            long start = System.currentTimeMillis();
            try {
                Thread.sleep(100); // simulate work
                System.out.println(Thread.currentThread().getName()
                        + " executed task [" + taskId + "] successfully, cost: "
                        + (System.currentTimeMillis() - start) + "ms");
            } catch (InterruptedException e) {
                Thread.currentThread().interrupt();
            }
        });
    }
}

Controller for load testing:

@RestController
@RequestMapping("/api/task")
public class TaskController {

    @Autowired
    private TaskService taskService;

    @PostMapping("/submit")
    public ResponseEntity<String> submit(@RequestParam String taskId) {
        taskService.submitTask(taskId);
        return ResponseEntity.ok("Task submitted");
    }
}

Advanced Recommendations

Replace the thread pool with Disruptor or LMAX RingBuffer for sub‑millisecond, ultra‑high‑throughput scenarios (up to millions of QPS).

Asynchronous merging + batch processing, e.g., collect tasks every 100 ms and process them together, possibly using ScheduledExecutorService or reactive frameworks like RxJava or Project Reactor.

Rate limiting, circuit breaking, and degradation using Sentinel or Resilience4j to protect the pool.

Conclusion

In a scenario of 100 k QPS and 100 ms task latency, thread‑pool optimization is essential. By aligning pool parameters with business characteristics, controlling concurrency, improving per‑thread efficiency, handling rejections gracefully, and employing asynchronous batch processing, the system achieves stable, high‑throughput performance.