How to Quickly Identify and Fix High CPU Usage in a Java Data Platform

Problem Background

An alert showed a data‑platform server’s CPU usage at 98.94% with sustained levels above 70%, despite the application not being high‑concurrency or CPU‑intensive, suggesting a code‑level issue rather than a hardware bottleneck.

Investigation Steps

2.1 Locate High‑Load Process (PID)

Log into the server and run top to view current load and identify the process with the highest CPU share. On an 8‑core machine, the load average confirmed high load, and PID 682 was observed to consume a large CPU percentage.

2.2 Identify the Business Service

Use pwdx 682 to reveal the executable path, confirming the process belongs to the data‑platform’s web service.

2.3 Find the Abnormal Thread and Code Line

Traditional four‑step method:

Sort processes by load: top -b -o %CPU (or similar) to find maxLoad(pid).

Show threads of the PID: top -Hp 682 to locate the heavy‑load thread ID.

Convert thread ID to hex: printf "0x%x\n" 1073.

Search the thread in a Java stack dump: jstack 682 | vim +/0x431 -.

Because the manual steps are time‑consuming, the author references a community script show-busy-java-threads.sh that automates the process.

Root‑Cause Analysis

The investigation traced the high CPU to a time‑utility method that converts timestamps to formatted date strings. The method is called repeatedly in a real‑time reporting query, calculating the number of seconds from midnight to the current time for every record, leading to millions of executions per query.

Abnormal method logic: Converts a timestamp to a specific date‑time format.

Upper‑layer call: Computes seconds for every second of the day up to now, stores results in a Set, and returns it.

Logic layer: Used by the data‑platform’s real‑time report queries, which invoke the method multiple times per query (n times), causing linear growth in CPU usage as the day progresses.

Solution

After confirming the method’s inefficiency, the code was replaced with a simpler calculation: directly compute currentSecond - midnightSecond without formatting or set creation. The new implementation reduced the number of calculations dramatically, and after deployment the server’s CPU load dropped by about 30×, returning to normal levels.

Key Takeaways

Performance optimization should accompany functional implementation; efficient code is a core engineering skill.

Conduct thorough code reviews and consider alternative implementations.

Never ignore small details in production issues; meticulous analysis leads to significant improvements.