Boosting Java Service Throughput from 50/s to 500/s: My Performance Tuning Journey

A ToB system with low concurrency suddenly needed to support at least 500 requests per second for a major client. Initial calculations suggested Tomcat with 100 threads could handle the load, but a load test showed only 50 /s throughput.

1. Analysis Process

Identify the “slow” cause

Ignore the high CPU usage for now.

The response time distribution revealed:

Minimum latency < 100 ms (good).

Maximum latency up to 5 s under load (unacceptable).

Most requests clustered around 4 s.

Thus the single‑node throughput was far below the target.

Locate the bottlenecks

Key suspects were:

Locks (synchronization, distributed, DB).

Time‑consuming operations (network calls, SQL).

We added latency‑based alerts:

Log a warning if an interface response exceeds 500 ms.

Log a warning if a remote call exceeds 200 ms.

Log a warning if Redis access exceeds 10 ms.

Log a warning if a SQL execution exceeds 100 ms.

After enabling these metrics, a slow SQL statement was identified:

update table set field = field - 1 where type = 1 and filed > 1;

The statement caused lock contention, accounting for over 80% of the total latency. The quick fix was to make the operation asynchronous, which reduced the maximum latency from 5 s to 2 s and the 95th‑percentile from 4 s to 1 s, roughly doubling throughput.

Further investigation

Log analysis showed occasional 100‑ms gaps between INFO entries, suggesting thread switches, excessive logging, or stop‑the‑world pauses. Actions taken:

Raised log level to DEBUG (minor impact).

Reconfigured @Async thread pools, limiting total core threads to ~50.

Increased JVM heap from 512 MB to 4 GB, reducing Young GC frequency.

These changes yielded another ~50% improvement, but CPU usage remained high.

Diagnosing high CPU usage

Investigation revealed frequent calls to BeanUtils.getBean() for a prototype‑scoped Redis helper, causing repeated createBean invocations and lock contention: RedisTool redisTool = BeanUtils.getBean(RedisMaster.class); The Redis helper class:

@Component
@Scope("prototype")
public class RedisMaster implements IRedisTool {
    // ...
}

Because Jedis is not thread‑safe, each request created a new Redis instance, triggering ~10 bean creations per request and inflating latency.

The fix was to replace prototype beans with direct new instantiation.

Additional timing utilities ( System.currentTimeMillis() and Hutool StopWatch) were also identified as non‑negligible under high concurrency.

2. Summary

The final improvements stemmed from:

Database tuning (buffer pool, redo log, etc.).

Code changes (async execution, thread‑pool tuning, Tomcat/Druid configuration).

JVM adjustments (memory size, GC settings).

Key commands used during troubleshooting: top -Hp <pid> – view per‑thread CPU usage. jstat -gc <pid> 2000 – monitor GC activity. jstack -l <pid> > stack.log – capture stack traces.

The experience highlighted the importance of systematic profiling, avoiding prototype‑scoped beans for high‑frequency operations, and keeping logging overhead low.

TODO

Further explore why createBean incurs such overhead and under what conditions Spring’s prototype scope is justified.