Spring Boot Tomcat Tuning: Breaking the 4K QPS Ceiling with Thread Pool & Keep-Alive Config

1. Why Throughput Stalls Despite Idle Resources

The author describes a real-world scenario: an 8C16G server running Spring Boot 2.x with embedded Tomcat, testing a product detail API averaging 50 ms latency under 1,000 concurrent users with a target of 10,000 QPS. With default settings the results were stuck at ~4,100 QPS, average response time 245 ms, P99 near 800 ms, 0.8% error rate, while CPU sat at only 25%. The bottleneck was Tomcat's default maxThreads=200, meaning only 200 requests can execute simultaneously. At 50 ms per request, 200 threads yield a theoretical maximum of 200 / 0.05 = 4,000 QPS — exactly the observed ceiling. The remaining requests queue up, so business logic never runs and CPU stays idle.

2. Tomcat NIO Network Model

Spring Boot 2.x uses NIO by default. The flow: Acceptor accepts TCP connections, registers them with a Poller that uses a Selector to detect readable/writable sockets. Once a full HTTP request is parsed, the Poller wraps it as a task and submits it to the worker thread pool (Executor) for business logic. Connector controls how many connections can be established; Executor controls how many requests run concurrently. These are independent — thousands of idle Keep-Alive connections do not consume worker threads, but they do consume connection slots.

3. Six Critical Parameters

In application.yml the following six settings govern concurrency:

maxThreads (default 200): maximum worker threads. The most important knob. Calculate via Little's Law (see below). Not "bigger is better" — excessive threads cause context-switch overhead.

minSpareThreads (default 10): threads pre-created at startup. For bursty traffic, raise to ~100 to avoid thread-creation latency.

maxConnections (default 8,192): maximum simultaneous TCP connections. Idle Keep-Alive connections count here. Set high enough for expected peak connections.

acceptCount (default 100): OS kernel backlog queue length when maxConnections is reached. Too small → connection refusals under burst.

connectionTimeout (default 20,000 ms): time waiting for client to send a complete request. Reduce to 5,000 ms to drop slow/malicious half-open connections.

keepAliveTimeout (defaults to connectionTimeout): idle Keep-Alive connection timeout. Tune to client reuse pattern (e.g., 30 s for steady pools, 5–10 s to reclaim idle slots).

Scope summary: acceptCount = kernel "waiting room" maxConnections = established connection ceiling maxThreads = actual workers keepAliveTimeout = how long idle connections may linger

4. Estimating Threads and Connections with Little's Law

Concurrent Threads = QPS × Average Response Time (RT)

For 10,000 QPS × 0.05 s = 500 threads. Add safety margin for GC, network jitter, queuing → author set maxThreads=800.

Connection estimation accounts for Keep-Alive reuse interval. If a request takes 50 ms and the client sends the next request 100 ms later, one connection is reused every 150 ms. Required connections ≈ 10,000 × 0.15 = 1,500. With sparse or bursty clients, actual connections can be higher, so keepAliveTimeout must prevent idle connection buildup.

5. Establishing a Reliable Load-Test Baseline

Always test with HTTP Keep-Alive enabled; otherwise you measure connection-setup overhead, not application throughput. Example ab command:

ab -n 200000 -c 1000 -k http://localhost:8080/api/product/detail

The -k flag enables Keep-Alive. In JMeter, check "Use KeepAlive" on the HTTP Request sampler. Start at moderate concurrency (e.g., 500) and ramp up to find the system's knee point.

6. Step-by-Step Tuning Walkthrough

Baseline (defaults): QPS 4,100, avg RT 245 ms, P99 800 ms, errors 0.8%, CPU 25%.

First adjustment:

server:
  tomcat:
    threads:
      max: 500
      min-spare: 100
    accept-count: 500

Result: QPS ~8,500, avg RT 118 ms, P99 240 ms, errors 0.01%. Still short of 10K QPS.

Analysis: Connection-layer bottleneck suspected. acceptCount too small for burst; idle Keep-Alive connections hogging slots. Second adjustment:

server:
  tomcat:
    threads:
      max: 800
      min-spare: 100
    accept-count: 1000
    max-connections: 20000
    connection-timeout: 5000
    keep-alive-timeout: 30000

Result: QPS stable at 10,300, avg RT 97 ms, P99 145 ms, errors 0%, CPU ~70% (healthy utilization).

Over-tuning test: Raising maxThreads to 2,000 on 8 cores degraded P99 due to context-switch thrashing and stack memory pressure (2,000 threads × 1 MB stack ≈ 2 GB). Parameters must be tuned incrementally, one variable at a time, guided by load-test data.

7. Production Monitoring Is Mandatory

Spring Boot Actuator + Micrometer exposes Tomcat metrics via Prometheus endpoint. Enable:

management:
  endpoints:
    web:
      exposure:
        include: health,metrics,prometheus

Key metrics (Prometheus names use underscores): tomcat_threads_busy — busy worker threads tomcat_threads_config_max — configured maxThreads tomcat_connections_current — current open connections tomcat_global_request_max — slowest request latency

Useful PromQL:

# Thread pool utilization
 tomcat_threads_busy / tomcat_threads_config_max

# Current connections (if maxConnections=20000)
 tomcat_connections_current

Alert if utilization > 0.8 for five minutes. Investigate whether RT increased or traffic grew. JMX alternative: Catalina:type=ThreadPool,name=http-nio-8080, Catalina:type=Connector,name=http-nio-8080, Catalina:type=GlobalRequestProcessor,name=http-nio-8080.

8. Common Pitfalls

a. Synchronous Blocking I/O in Business Code

If handlers perform blocking JDBC, HTTP, or Redis calls, threads spend most time waiting. Scaling threads from 200 to 2,000 only increases memory (2 GB stack) and context-switch cost; downstream connection pools also saturate. Fix: reduce business RT via async calls, batching, or reactive patterns before tuning Tomcat.

b. Keep-Alive Connections Exhausting maxConnections

Clients may open many connections and leave them idle. Post-tuning monitoring showed 1,000–2,000 idle connections. Reducing keepAliveTimeout to 10 s reclaimed slots. If a reverse proxy (Nginx) handles connection reuse, backend Tomcat pressure drops further. Caution: too short a timeout forces frequent reconnects if client think-time exceeds the timeout.

c. File Descriptor Limits

High connection counts trigger "Too many open files". Raise ulimit -n to 100,000+. Use ss -s to inspect TCP states: many CLOSE_WAIT indicates application not closing sockets; many TIME_WAIT suggests client-side short connections without Keep-Alive — fix on client side.

d. Bigger Parameters Are Not Better

Excessive maxThreads causes context-switch storms and GC pressure. Excessive acceptCount lets queues grow until clients time out, worsening tail latency. Tuning loop: estimate maxThreads via Little's Law → load test → if QPS misses target and thread pool saturated, increase slightly → if connection-layer anomalies appear, adjust maxConnections / acceptCount → set keepAliveTimeout based on observed idle connection count → monitor continuously.

There is no universal optimal value; it depends on business RT, client behavior, OS, and hardware. The reusable process is: estimate → load test to find knee point → monitor in production → iterate.