Why Does a ThreadPool Queue Tasks Even When Max Threads Are Idle?

When developers set corePoolSize to 5, maximumPoolSize to 10, and a queue capacity of 100, they often expect the sixth task to spawn a new thread immediately. In reality, the task is queued, and only after the queue fills does the pool create a non‑core thread.

How the thread pool runs tasks

Core threads not full – a new thread is created to run the task.

Core threads full – the task is placed into the work queue.

Queue full – if the current pool size is still below maximumPoolSize, a temporary (non‑core) thread is created.

Queue full and max threads reached – the rejection policy is applied.

Why the queue comes before max threads

The pool is designed as a rate‑limiter and buffer , not to instantly saturate the CPU. Creating a thread is heavyweight: each thread needs at least 1 MB of stack memory and a system‑level call, which can cause CPU spikes and even OOM when thousands of threads are created simultaneously.

Having many threads does not guarantee higher throughput because the number of CPU cores is limited; excessive threads lead to frequent context switches, wasting CPU cycles on saving and restoring thread state.

Therefore, using a queue as a buffer allows the pool to handle burst traffic with minimal resource overhead.

Can we create max threads first?

For background jobs or batch processing, the default JDK logic works well. However, for latency‑sensitive web servers like Tomcat, queuing a request while CPU cores are idle results in poor user experience.

Tomcat prefers to create a thread as soon as possible and only falls back to queuing when thread creation is impossible.

Tomcat's implementation

Tomcat does not rewrite ThreadPoolExecutor; it subclasses LinkedBlockingQueue to create a TaskQueue with a custom offer method.

// JDK ThreadPoolExecutor snippet
if (isRunning(c) && workQueue.offer(command)) {
    // task queued, core threads will consume it
} else if (addWorker(command, false)) {
    // queue full, try to create a non‑core thread
}

@Override
public boolean offer(Runnable o) {
    // 1. If pool size equals max, just queue
    if (parent.getPoolSize() == parent.getMaximumPoolSize()) {
        return super.offer(o);
    }
    // 2. If submitted tasks <= current pool size, queue
    if (parent.getSubmittedCount() <= parent.getPoolSize()) {
        return super.offer(o);
    }
    // 3. If pool size < max, force thread creation by returning false
    if (parent.getPoolSize() < parent.getMaximumPoolSize()) {
        return false; // trick executor into thinking queue is full
    }
    // default: queue
    return super.offer(o);
}

This logic makes Tomcat follow the order: core full → create max thread → max full → queue → queue full → reject.

Final notes

Using new LinkedBlockingQueue() without a size argument creates an unbounded queue (capacity Integer.MAX_VALUE), effectively rendering maximumPoolSize useless because tasks never fill the queue.

Configure the pool based on the workload:

High‑concurrency, low‑latency services – adopt Tomcat's queue‑first‑then‑thread strategy.

Background batch jobs – keep the JDK default with a reasonably sized bounded queue.