How I Reduced Helios API Latency from Seconds to Milliseconds with Arthas

Background

The Helios system processes massive daily score data, fetching up to 1440 minutes of scores for each service, resulting in API latency of several seconds. Initial profiling showed that while database queries took only ~11 ms, the data assembly layer consumed most of the time.

Optimization Process

Initial Unoptimized Version

private HeliosGetScoreResponse queryScores(HeliosGetScoreRequest request) { ... }

Arthas Trace

---ts=2021-08-17 ...
---[4000ms] xxxService.controller.HeliosController:queryScores()
   +---[... many method calls ...]

Analysis

Arthas reported a total of ~4 seconds, but the actual network latency was ~350‑450 ms; the extra time came from the tracing overhead itself, especially when profiling loops.

The method contains three nested loops: outer loop over ~140 appIds, middle loop over merged data (usually 1 entry per day), and inner loop over 1440 minutes.

The most expensive operation was SimpleDateFormat.formatDate().

First Optimization

Optimization Direction

Change the iteration strategy to avoid creating tens of thousands of objects.

Replace Set<String> dateSet with Set<Date> to reduce repeated formatDate() calls.

Attempt to replace string‑to‑int parsing with a pre‑built map (later found Integer.parseInt was still fastest).

Code

private HeliosGetScoreResponse queryScores(HeliosGetScoreRequest request) {
    List<HeliosScore> heliosScoresRecord = heliosService.queryScoresTimeBetween(...);
    if (CollectionUtils.isEmpty(heliosScoresRecord)) return response;
    Set<Date> dateSet = new HashSet<>();
    List<HeliosScore> heliosScores = HeliosDataMergeJob.mergeData(heliosScoresRecord);
    Map<String, List<HeliosScore>> groupByAppId = heliosScores.stream()
        .collect(Collectors.groupingBy(HeliosScore::getAppId));
    for (List<HeliosScore> scores : groupByAppId.values()) {
        // ... simplified loop using Date objects and reduced parsing ...
    }
    response.setDates(new ArrayList<>(dateSet).stream()
        .sorted()
        .map(DateUtils.yyyyMMddHHmm::formatDate)
        .collect(Collectors.toList()));
    return response;
}

Arthas Trace

... execution time reduced by ~50ms ...

Analysis

The next biggest cost was Date.compareTo inside the loop and repeated getter calls.

Second Optimization

Optimization Direction

Replace Date objects with long timestamps for comparisons.

Batch set timestamps instead of updating each iteration.

Insert dates into the set only once.

Pre‑allocate list sizes to avoid repeated resizing.

Code

private HeliosGetScoreResponse queryScores(HeliosGetScoreRequest request) {
    // use long indexDateMills, heliosScoreTimeFromMills, etc.
    // while loops compare primitive longs
    // dateSet.add(new Date(indexDateMills)) only on first iteration
    // score.setScores(new ArrayList<>(estimatedSize));
}

Arthas Trace

... execution time reduced by ~80ms, remaining ~60ms ...

Analysis

Remaining hot spots were simple getter calls ( list.size(), list.get()) and list additions, which still contributed noticeable overhead when executed many times.

Third Optimization

Optimization Direction

Minimize list property accesses.

Replace repeated list.add with a single list.addAll using subList.

Code

while (indexDateMills <= requestEnd && ... ) {
    // only collect indices, then after loop:
    score.getScores().addAll(scoreIntList.subList(start, end));
}

Arthas Trace

... execution time reduced by another ~100ms, now ~60ms total ...

Fourth Optimization

Optimization Direction

Fix SQL to avoid fetching duplicate rows.

Skip merge logic when only a single record is returned.

Arthas Trace

... database query now dominates at 25‑40ms ...

Result

After all optimizations, the API latency dropped from several seconds to roughly 60 ms, with the database query being the remaining bottleneck.

Conclusion

Avoid unnecessary object creation. SimpleDateFormat is expensive. Date.compareTo adds noticeable cost.

Even trivial operations like list.size() and list.add accumulate when called millions of times.

Profiling tools such as Arthas are essential to identify true performance hotspots.