How to Diagnose High CPU Usage in a Linux Gateway Process

Background

A customer reported that after a product upgrade their system became sluggish and CPU usage surged. The gateway process (PID 14094) was consuming an abnormal 891% CPU, which is the sum of its threads' usage.

1. Identify the high‑CPU process

Run # top to see which process has the highest CPU consumption.

2. Inspect thread‑level CPU usage

Use # top -H -p 14094 to list all threads of the process. Although 107 threads appear, nine of them dominate the CPU, with thread 14086 (LWP 14086, thread 37) being the worst offender.

3. Capture stack traces of threads

Execute # gstack 14094 > gstack.log and search the log for the problematic thread ID (14086). The log shows two frames for thread 37:

#0  0x40000410 in __kernel_vsyscall ()
#1  0x40241f33 in poll () from /lib/i686/nosegneg/libc.so.6

4. Dump the process image

Run # gcore 14094 to generate a core file named core.14094. This file contains the full memory image of the process.

5. Trace system calls

Use # strace -T -r -c -p 14094. The -c option provides a summary of time spent in each system call; omitting -c shows per‑call timing and return values.

6. Debug the core dump

Load the core file in gdb and switch to the offending thread:

(gdb) gdb gateway core.14094
(gdb) thread 37
(gdb) where
#0  0x40000410 in __kernel_vsyscall ()
#1  0x40241f33 in poll () from /lib/i686/nosegneg/libc.so.6

With the detailed stack you can inspect variables and correlate the high CPU usage to the poll() call.

Conclusion

The analysis flow—process ID → thread ID → thread stack → system‑call timing → source‑code inspection—provides a repeatable method for diagnosing CPU‑bound issues in Linux services. The actual source code was omitted for proprietary reasons, but the steps and commands are fully applicable to similar problems.