How to Quickly Diagnose and Fix High CPU Usage on a Data Platform Server

1. Problem Background

Yesterday afternoon an ops alert showed CPU usage at 98.94% on a data‑platform server, staying above 70% for a while. The service is not high‑concurrency or CPU‑intensive, so the spike likely originates from problematic business code rather than hardware limits.

2. Investigation Steps

2.1 Identify High‑Load Process (PID)

Log into the server and run top to view load average and process list. The load average on an 8‑core machine confirms high load, and PID 682 shows a large CPU share.

2.2 Locate the Abnormal Business Service

Use pwdx 682 to find the process directory, which points to the data‑platform web service.

2.3 Find the Faulty Thread and Code Line

Traditional four‑step method:

Sort processes by load: top -o %CPU -p ... Show threads of the PID: top -Hp 682 Convert thread ID to hex: printf "0x%x\n" $TID Search jstack output: jstack 682 | vim +/0x... - To speed up, use the script show-busy-java-threads.sh which automates the above.

The analysis revealed a time‑utility method consuming excessive CPU. The method converts timestamps to formatted date strings and is called repeatedly in real‑time report queries.

3. Root Cause Analysis

The method is invoked for every second from midnight to the current time, leading to 36 000 × n calls per query, and the number grows linearly toward midnight. Heavy use in real‑time queries and alerts caused the CPU spike.

Problematic logic: converting timestamps to date strings.

Upper‑level call: calculates seconds from midnight to now and stores results in a set.

Business layer: real‑time report queries invoke the method many times per request.

4. Solution

Replace the costly conversion with a simpler calculation (current seconds – midnight seconds) and stop using the returned set. After deploying the change, CPU usage dropped by a factor of 30, returning to normal levels.

5. Takeaways

Performance optimization is as important as functional correctness.

Conduct code reviews and consider more efficient implementations.

Never ignore small details in production incidents; thorough investigation drives growth.

References

线上服务 CPU 100%？一键定位 so easy！

linux 系统监控、诊断工具之 top 详解