How to Size Thread Pools for Better CPU Utilization and QPS

1. Introduction

When an online service experiences severe timeout and "too many connections" errors, the usual quick fix is to increase the connection count and thread‑pool size, then restart the service. This approach only treats the symptom because the root cause is unknown.

A common misconception is that a larger thread pool automatically makes requests faster, assuming the pool was too small.

This article discusses how to set thread‑pool size reasonably.

2. Problem

Assume there are two tasks, A and B.

Option 1: One thread executes A then B.
Option 2: Two threads execute A and B concurrently.

Many would say Option 2 is faster because the tasks run in parallel. Before answering, we need to review how threads actually work.

3. Thread Execution

CPU schedules threads, and a single CPU core can execute only one thread at any moment. What appears as concurrent execution is achieved by time‑slice rotation.

The operating system gives each task a time slice, saves its state, and switches to the next task. This state‑save‑and‑restore process is called a context switch, and it consumes time.

Therefore, some may argue that Option 1 is faster because it avoids thread switching, while Option 2 incurs extra switching overhead.

Why do we need multithreading at all? The answer lies in real‑world workloads.

4. Why Use Multithreading

Typical request processing flow:

1. Initiate network request 2. Web server parses request 3. Backend database query 4. Process retrieved data 5. Return result to user

This flow involves three types of work:

Network I/O → network I/O Request parsing → CPU Database query → network I/O MySQL data fetch → disk I/O Data processing → CPU Response transmission → network I/O

Only the parsing and data‑processing steps actually use the CPU; the rest are I/O‑bound. While a thread waits for I/O, the CPU stays idle, leading to low CPU utilization. Multithreading allows other threads to use the CPU during these idle periods, improving overall utilization.

5. Improving QPS/TPS

System performance is often measured by QPS/TPS (queries/transactions per second).

QPS/TPS = number of requests per second Concurrency = number of simultaneous requests Response time = average time to handle a request

Since QPS = concurrency / response time, increasing concurrency (e.g., by enlarging the thread pool) can raise QPS, but only if response time does not increase significantly. Larger thread pools cause more context switches, which can increase response time.

6. How to Set the Thread Count

A commonly used formula is:

OptimalThreadCount = ((ThreadWaitTime + ThreadCPUTime) / ThreadCPUTime) * CPUCount

For example, with a 4‑core CPU, a thread that spends 20 ms on CPU and 80 ms waiting for I/O yields: ((80 + 20) / 20) * 4 = 20 threads The larger the waiting time, the more threads are needed to keep the CPU busy. However, the exact number should be validated with load testing in the production environment.

7. Basic Guidelines

When many services share a single pool, start with these rule‑of‑thumb values (adjust after testing):

CPU‑bound workloads: threads ≈ CPU cores + 1 or CPU cores × 2 (e.g., 5–8 for 4 cores). I/O‑bound workloads: threads ≈ CPU cores / (1 – 0.9) (e.g., about 40 for 4 cores).

8. Conclusion

Thread‑pool sizing should be based on analysis of CPU vs I/O bottlenecks rather than blindly increasing the pool size. Understanding context switches and CPU utilization helps avoid common pitfalls and leads to better QPS/TPS performance. Further reading on Redis and Nginx can deepen this knowledge.