How I Reduced a 4‑Second Java API Call to 60ms with Arthas Tracing

Background

Helios system processes large amounts of data; querying a day's scores for all services returned hundreds of thousands of points and the API latency could reach several seconds.

Optimization Process

Initial Unoptimized Version

private HeliosGetScoreResponse queryScores(HeliosGetScoreRequest request) {
    HeliosGetScoreResponse response = new HeliosGetScoreResponse();
    List<HeliosScore> heliosScores = heliosService.queryScoresTimeBetween(request.getStartTime(), request.getEndTime(), request.getFilterByAppId());
    if (CollectionUtils.isEmpty(heliosScores)) {
        return response;
    }
    Set<String> dateSet = new HashSet<>();
    Map<String, List<HeliosScore>> groupByAppIdHeliosScores = heliosScores.stream().collect(Collectors.groupingBy(HeliosScore::getAppId));
    for (List<HeliosScore> value : groupByAppIdHeliosScores.values()) {
        value.sort(Comparator.comparing(HeliosScore::getTimeFrom));
        HeliosGetScoreResponse.Score score = new HeliosGetScoreResponse.Score();
        score.setNamespace(value.get(0).getNamespace());
        score.setAppId(value.get(0).getAppId());
        for (HeliosScore heliosScore : value) {
            List<HeliosScore> splitHeliosScores = heliosScore.split();
            for (HeliosScore splitHeliosScore : splitHeliosScores) {
                if (splitHeliosScore.getTimeFrom().compareTo(request.getStartTime()) < 0) {
                    continue;
                }
                if (splitHeliosScore.getTimeFrom().compareTo(request.getEndTime()) > 0) {
                    break;
                }
                dateSet.add(DateUtils.yyyyMMddHHmm.formatDate(splitHeliosScore.getTimeFrom()));
                if (splitHeliosScore.getScores() == null) {
                    splitHeliosScore.setScores("100");
                    log.error("Missing data: {}", heliosScore);
                }
                score.add(Math.max(0, Integer.parseInt(splitHeliosScore.getScores())), null);
            }
        }
        response.getValues().add(score);
    }
    response.setDates(new ArrayList<>(dateSet).stream().sorted().collect(Collectors.toList()));
    return response;
}

Arthas trace showed the method took about 4 seconds, while the raw network latency was only 350‑450 ms; most of the time was spent in the Java code.

First Optimization

Changes:

Iterate over time points differently, merging large objects into smaller ones.

Replace Set<String> dateSet with Set<Date> to avoid repeated formatDate() calls.

Cache integer parsing by using a pre‑built map (later found Integer.parseInt was still fastest).

private HeliosGetScoreResponse queryScores(HeliosGetScoreRequest request) {
    HeliosGetScoreResponse response = new HeliosGetScoreResponse();
    List<HeliosScore> heliosScoresRecord = heliosService.queryScoresTimeBetween(request.getStartTime(), request.getEndTime(), request.getFilterByAppId());
    if (CollectionUtils.isEmpty(heliosScoresRecord)) {
        return response;
    }
    Set<Date> dateSet = new HashSet<>();
    List<HeliosScore> heliosScores = HeliosDataMergeJob.mergeData(heliosScoresRecord);
    Map<String, List<HeliosScore>> groupByAppIdHeliosScores = heliosScores.stream().collect(Collectors.groupingBy(HeliosScore::getAppId));
    for (List<HeliosScore> scores : groupByAppIdHeliosScores.values()) {
        HeliosScore heliosScore = scores.get(0);
        HeliosGetScoreResponse.Score score = new HeliosGetScoreResponse.Score();
        score.setNamespace(heliosScore.getNamespace());
        score.setAppId(heliosScore.getAppId());
        score.setScores(new ArrayList<>());
        response.getValues().add(score);
        List<Integer> scoreIntList = HeliosHelper.splitScores(heliosScore);
        Calendar indexDate = DateUtils.roundDownMinute(request.getStartTime().getTime());
        int index = 0;
        while (indexDate.getTime().compareTo(heliosScore.getTimeFrom()) > 0) {
            heliosScore.getTimeFrom().setTime(heliosScore.getTimeFrom().getTime() + 60_000);
            index++;
        }
        while (indexDate.getTime().compareTo(request.getEndTime()) <= 0 && indexDate.getTime().compareTo(heliosScore.getTimeTo()) <= 0 && index < scoreIntList.size()) {
            Integer scoreInt = scoreIntList.get(index++);
            score.getScores().add(scoreInt);
            dateSet.add(new Date(indexDate.getTime()));
            indexDate.add(Calendar.MINUTE, 1);
        }
        response.setDates(new ArrayList<>(dateSet).stream().sorted().map(DateUtils.yyyyMMddHHmm::formatDate).collect(Collectors.toList()));
        return response;
    }
    return response;
}

This reduced the execution time by about 50 ms.

Second Optimization

Further changes:

Replace Date objects with long timestamps for comparisons.

Perform timestamp arithmetic directly, setting the Date only once.

Insert dates into the set only once using a flag.

Pre‑size the score list based on expected size.

private HeliosGetScoreResponse queryScores(HeliosGetScoreRequest request) {
    HeliosGetScoreResponse response = new HeliosGetScoreResponse();
    List<HeliosScore> heliosScoresRecord = heliosService.queryScoresTimeBetween(request.getStartTime(), request.getEndTime(), request.getFilterByAppId());
    if (CollectionUtils.isEmpty(heliosScoresRecord)) {
        return response;
    }
    Set<Date> dateSet = new HashSet<>();
    boolean isDateSetInitial = false;
    int scoreSize = 16;
    List<HeliosScore> heliosScores = HeliosDataMergeJob.mergeData(heliosScoresRecord);
    Map<String, List<HeliosScore>> groupByAppIdHeliosScores = heliosScores.stream().collect(Collectors.groupingBy(HeliosScore::getAppId));
    for (List<HeliosScore> scores : groupByAppIdHeliosScores.values()) {
        HeliosScore heliosScore = scores.get(0);
        HeliosGetScoreResponse.Score score = new HeliosGetScoreResponse.Score();
        score.setNamespace(heliosScore.getNamespace());
        score.setAppId(heliosScore.getAppId());
        score.setScores(new ArrayList<>(scoreSize));
        response.getValues().add(score);
        List<Integer> scoreIntList = HeliosHelper.splitScores(heliosScore);
        long indexDateMills = request.getStartTime().getTime();
        int index = 0;
        long heliosScoreTimeFromMills = heliosScore.getTimeFrom().getTime();
        while (indexDateMills > heliosScoreTimeFromMills) {
            heliosScoreTimeFromMills += 60_000;
            index++;
        }
        heliosScore.getTimeFrom().setTime(heliosScoreTimeFromMills);
        long requestEndTimeMills = request.getEndTime().getTime();
        long heliosScoreTimeToMills = heliosScore.getTimeTo().getTime();
        while (indexDateMills <= requestEndTimeMills && indexDateMills <= heliosScoreTimeToMills && index < scoreIntList.size()) {
            score.getScores().add(scoreIntList.get(index++));
            if (!isDateSetInitial) {
                dateSet.add(new Date(indexDateMills));
            }
            indexDateMills += 60_000;
        }
        isDateSetInitial = true;
        scoreSize = (int) (score.getScores().size() * 1.1);
    }
    response.setDates(new ArrayList<>(dateSet).stream().sorted().map(DateUtils.yyyyMMddHHmm::formatDate).collect(Collectors.toList()));
    return response;
}

This step saved another ~80 ms, leaving about 160 ms.

Third Optimization

Further reductions:

Avoid repeated list.get and list.size calls inside loops.

Collect scores with subList and addAll in bulk.

while (indexDateMills <= requestEndTimeMills && indexDateMills <= heliosScoreTimeToMills && index++ < scoreIntListSize) {
    if (!isDateSetInitial) {
        dateSet.add(new Date(indexDateMills));
    }
    indexDateMills += 60_000;
}
score.getScores().addAll(scoreIntList.subList(indexStart, index - 1));

This optimization shaved roughly 100 ms, bringing the total to about 60 ms.

Fourth Optimization

Finally, the SQL query was tightened to avoid fetching extra rows, and the merge step was bypassed for single‑record cases.

After these changes the database query itself became the dominant cost (≈25‑40 ms), and the overall API latency settled around 60 ms.

Conclusion

Key take‑aways:

Minimize object creation, especially in hot loops.

Avoid SimpleDateFormat and frequent Date.compareTo calls.

Even trivial methods like list.size() or list.add add up when called millions of times.

Profiling tools such as Arthas are essential to identify real bottlenecks.