How to Fix Common Java Backend Performance Pitfalls: Deadlocks, Thread Pools, and Logging

Many architecture articles focus on high‑level design, but few address code‑level performance optimization; this is akin to building a house with a solid frame but unskilled workers, leading to leaks and cracks.

Server Environment

Server configuration: 4‑core CPU, 8 GB RAM, 4 servers

Message queue: RabbitMQ

Database: DB2

SOA framework: internally packaged Dubbo

Cache: Redis, Memcached

Configuration management: internal system

Problem Description

Single instance handles 40 TPS; scaling to four servers only reaches 60 TPS, showing poor scalability.

Frequent database deadlocks cause complete service outages.

Improper use of transactions leads to excessively long transaction times.

Production environments often encounter memory overflow and CPU saturation.

Poor fault tolerance means a small bug can render the service unavailable.

Critical logs are missing or contain useless information.

Static configuration data is repeatedly read from the database, causing high I/O.

Incomplete project separation results in multiple WAR packages deployed in a single Tomcat.

Platform bugs or feature defects reduce availability.

Lack of rate‑limiting allows VIP merchants to stress‑test the production environment.

No fallback strategy leads to long recovery times or brute‑force rollbacks.

Absence of proper monitoring prevents early detection of bottlenecks.

Optimization Solutions

1. Database Deadlock Mitigation

Example of a deadlock situation is shown below:

The deadlock occurs because sessions A and B wait for each other, often caused by mixing many FOR UPDATE statements with transactions, leading to Record Lock, Gap Lock, and Next‑Key Lock.

Instead of pessimistic locking for idempotency, adopt one of the following:

Use Redis distributed locks with sharding; a failed node can be reclaimed.

Apply primary‑key based idempotency: insert into a guard table and rely on duplicate‑key errors to reject repeats.

Implement a version‑number mechanism.

All approaches must set an expiration time so that locks are released when they time out.

2. Reduce Transaction Duration

Pseudocode illustration of a long‑running transaction:

Embedding operations such as HTTP client calls that may block for a long time inside a transaction dramatically extends its duration and reduces concurrency. The principle is “fast in, fast out”: keep transaction code minimal and extract blocking calls.

3. CPU Saturation Analysis

Three main causes were identified:

3.1 Database Connection Pool

Using C3P0 in high‑concurrency tests caused timeouts:

com.yeepay.g3.utils.common.exception.YeepayRuntimeException: Could not get JDBC Connection; nested exception is java.sql.SQLException: An attempt by a client to checkout a Connection has timed out.

C3P0 performs poorly under heavy load.

3.2 Thread‑Pool Misuse

Original code created an unbounded cached thread pool:

private static final ExecutorService executorService = Executors.newCachedThreadPool();

This spawned millions of threads, exhausting resources. Switching to a fixed pool reduced the issue but introduced an unbounded queue:

private static final ExecutorService executorService = Executors.newFixedThreadPool(50);

The unbounded queue allowed unlimited tasks to accumulate, leading to memory pressure.

Final thread‑pool strategy: either offload asynchronous tasks to a dedicated processor or adopt the Akka framework for actor‑based concurrency.

4. Logging Best Practices

Logs should be emitted via logger.error or logger.warn with a concise format, for example:

%d %-5p %c [%t] - %m%n

Over‑verbose logging increases disk I/O and can block threads; a proper format reduces contention and improves throughput.

By applying these optimizations—distributed locking, short transactions, correct thread‑pool sizing, and disciplined logging—the system’s throughput increased from 40 TPS to over 100 TPS and overall stability improved markedly.