How to Diagnose Slow OpenAPI Calls with TProfiler Without Changing Code

Background

Recently I was plagued by intermittent performance issues: an OpenAPI endpoint sometimes responded in tens of milliseconds, other times took several seconds.

Our OpenAPI response time is fast at times and slow at others, ranging from dozens of milliseconds to a few seconds.

Attempted Solution

The problem wasn't a business bug, so I tried to reproduce it in a test environment, but the test setup was too fast to trigger the issue. I even asked operations to check Nginx and OpenAPI response times, which confirmed the latency problem.

To understand the issue clearly, I outlined the call flow:

The architecture follows a typical layered design:

Client →

Nginx

Nginx

forwards to backend web service web service calls backend Service via

RPC

Log Approach

We first considered adding logs at slow methods, but the request chain is long, logging would require many changes, and a missed log could still hide the problem. Moreover, code changes would need a new deployment.

Tool Analysis

The best solution is to use a non‑intrusive profiling agent. I chose Alibaba’s open‑source TProfiler and added it via the JVM -javaagent option: -javaagent:/path/to/TProfiler/lib/tprofiler-1.0.jar This agent monitors method execution time without modifying code and has minimal performance impact.

Tool Usage

Steps to get started:

Clone the modified source and build it:

git clone https://github.com/crossoverJie/TProfiler

mvn assembly:assembly

The built JAR is located at TProfiler/pkg/TProfiler/lib/tprofiler-1.0.1.jar. Add this JAR to the JVM startup parameters and provide a configuration file.

#log file name
logFileName=tprofiler.log
#basic configuration items
startProfTime=1:00:00
endProfTime=23:00:00
eachProfUseTime=10
eachProfIntervalTime=1
samplerIntervalTime=20
port=50000
debugMode=false
needNanoTime=false
ignoreGetSetMethod=true
logFilePath=/data/work/logs/tprofile/${logFileName}
methodFilePath=/data/work/logs/tprofile/${methodFileName}
samplerFilePath=/data/work/logs/tprofile/${samplerFileName}
includePackageStartsWith=top.crossoverjie.cicada.example.action
excludePackageStartsWith=com.taobao.sketch;org.apache.velocity;com.alibaba.forest.domain.dataobject

After launching the application, TProfiler records method information in the configured directory.

To simulate the slow‑response issue, I created a simple HTTP endpoint using Cicada that calls two time‑consuming methods.

When the application runs, TProfiler writes method details to tmethod.log. If the log is empty, you can flush the latest samples with:

java -cp /TProfiler/tprofiler.jar com.taobao.profile.client.TProfilerClient 127.0.0.1:50000 flushmethod

The service exposed by TProfiler reads tprofile.log, parses it, and writes aggregated data to tmethod.log.

The port used is the one defined in the configuration file.

Opening tmethod.log shows entries like thread ID, stack depth, method ID, and elapsed time (ms). The first column is the method ID, which can be used to look up detailed timings in tprofile.log. The last column is the source line number.

Method Detail Timing

To see all timings for a specific method, first find its ID in tmethod.log, e.g., method selectDB() has ID 2:

2 top/crossoverjie/cicada/example/action/DemoAction:selectDB:84

Then filter the detailed log: grep 2 tprofiler.log This yields all execution records for that method.

Conclusion

Using TProfiler we discovered that the latency spikes were caused by database‑related methods under heavy DB load. The short‑term mitigation was to make write‑to‑DB operations asynchronous. For long‑term stability we plan to split hot and cold data, shard the database, and consider integrating a full‑featured APM tool such as Pinpoint.

Many profiling tools exist; choose the one that fits your stack. In environments without a distributed tracing system, a lightweight agent like TProfiler can provide valuable insight and can be extended with custom visualizations.