Performance Optimization of Helios Scoring Service Using Arthas Trace

The Helios system needed to process a large volume of rating data, returning scores for every minute of a day across many applications, which caused API response times of several seconds. Initial profiling showed that database queries took only about 11 ms, while most of the latency was incurred during data assembly in the application.

Optimization Process

Initial Unoptimized Version

private HeliosGetScoreResponse queryScores(HeliosGetScoreRequest request) {
    HeliosGetScoreResponse response = new HeliosGetScoreResponse();
    List
heliosScores = heliosService.queryScoresTimeBetween(request.getStartTime(), request.getEndTime(), request.getFilterByAppId());
    if (CollectionUtils.isEmpty(heliosScores)) {
        return response;
    }
    Set
dateSet = new HashSet<>();
    Map
> groupByAppIdHeliosScores = heliosScores.stream().collect(Collectors.groupingBy(HeliosScore::getAppId));
    for (List
value : groupByAppIdHeliosScores.values()) {
        value.sort(Comparator.comparing(HeliosScore::getTimeFrom));
        HeliosGetScoreResponse.Score score = new HeliosGetScoreResponse.Score();
        score.setNamespace(value.get(0).getNamespace());
        score.setAppId(value.get(0).getAppId());
        for (HeliosScore heliosScore : value) {
            List
splitHeliosScores = heliosScore.split();
            for (HeliosScore splitHeliosScore : splitHeliosScores) {
                if (splitHeliosScore.getTimeFrom().compareTo(request.getStartTime()) < 0) {
                    continue;
                }
                if (splitHeliosScore.getTimeFrom().compareTo(request.getEndTime()) > 0) {
                    break;
                }
                dateSet.add(DateUtils.yyyyMMddHHmm.formatDate(splitHeliosScore.getTimeFrom()));
                if (splitHeliosScore.getScores() == null) {
                    splitHeliosScore.setScores("100");
                    log.error("查询时发现数据缺失: {}", heliosScore);
                }
                score.add(Math.max(0, Integer.parseInt(splitHeliosScore.getScores())), null);
            }
        }
        response.getValues().add(score);
    }
    response.setDates(new ArrayList<>(dateSet).stream().sorted().collect(Collectors.toList()));
    return response;
}

Arthas trace revealed that the method spent about 4 seconds overall, with most of the extra time caused by the tracing itself, especially due to heavy loops. The most expensive operations were SimpleDateFormat.formatDate() , repeated Date.compareTo , and frequent list.size() / list.get() calls.

First Optimization

Changed the date collection from Set<String> to Set<Date> to avoid repeated formatting, and prepared a map of string‑to‑integer values to reduce Integer.parseInt calls (later found parsing was still fastest).

private HeliosGetScoreResponse queryScores(HeliosGetScoreRequest request) {
    HeliosGetScoreResponse response = new HeliosGetScoreResponse();
    List
heliosScoresRecord = heliosService.queryScoresTimeBetween(request.getStartTime(), request.getEndTime(), request.getFilterByAppId());
    if (CollectionUtils.isEmpty(heliosScoresRecord)) {
        return response;
    }
    Set
dateSet = new HashSet<>();
    List
heliosScores = HeliosDataMergeJob.mergeData(heliosScoresRecord);
    Map
> groupByAppIdHeliosScores = heliosScores.stream().collect(Collectors.groupingBy(HeliosScore::getAppId));
    for (List
scores : groupByAppIdHeliosScores.values()) {
        HeliosScore heliosScore = scores.get(0);
        HeliosGetScoreResponse.Score score = new HeliosGetScoreResponse.Score();
        score.setNamespace(heliosScore.getNamespace());
        score.setAppId(heliosScore.getAppId());
        score.setScores(new ArrayList<>());
        response.getValues().add(score);
        List
scoreIntList = HeliosHelper.splitScores(heliosScore);
        Calendar indexDate = DateUtils.roundDownMinute(request.getStartTime().getTime());
        int index = 0;
        while (indexDate.getTime().compareTo(heliosScore.getTimeFrom()) > 0) {
            heliosScore.getTimeFrom().setTime(heliosScore.getTimeFrom().getTime() + 60_000);
            index++;
        }
        while (indexDate.getTime().compareTo(request.getEndTime()) <= 0 &&
               indexDate.getTime().compareTo(heliosScore.getTimeTo()) <= 0 &&
               index < scoreIntList.size()) {
            Integer scoreInt = scoreIntList.get(index++);
            score.getScores().add(scoreInt);
            dateSet.add(indexDate.getTime());
            indexDate.add(Calendar.MINUTE, 1);
        }
    }
    response.setDates(new ArrayList<>(dateSet).stream().sorted().map(DateUtils.yyyyMMddHHmm::formatDate).collect(Collectors.toList()));
    return response;
}

This change reduced the execution time by roughly 50 ms.

Second Optimization

Replaced Date objects with primitive long timestamps for comparisons, eliminated repeated setTime calls, and avoided inserting duplicate dates into the set.

private HeliosGetScoreResponse queryScores(HeliosGetScoreRequest request) {
    HeliosGetScoreResponse response = new HeliosGetScoreResponse();
    List
heliosScoresRecord = heliosService.queryScoresTimeBetween(request.getStartTime(), request.getEndTime(), request.getFilterByAppId());
    if (CollectionUtils.isEmpty(heliosScoresRecord)) {
        return response;
    }
    Set
dateSet = new HashSet<>();
    boolean isDateSetInitial = false;
    int scoreSize = 16;
    List
heliosScores = HeliosDataMergeJob.mergeData(heliosScoresRecord);
    Map
> groupByAppIdHeliosScores = heliosScores.stream().collect(Collectors.groupingBy(HeliosScore::getAppId));
    for (List
scores : groupByAppIdHeliosScores.values()) {
        HeliosScore heliosScore = scores.get(0);
        HeliosGetScoreResponse.Score score = new HeliosGetScoreResponse.Score();
        score.setNamespace(heliosScore.getNamespace());
        score.setAppId(heliosScore.getAppId());
        score.setScores(new ArrayList<>(scoreSize));
        response.getValues().add(score);
        List
scoreIntList = HeliosHelper.splitScores(heliosScore);
        long indexDateMills = request.getStartTime().getTime();
        int index = 0;
        long heliosScoreTimeFromMills = heliosScore.getTimeFrom().getTime();
        while (indexDateMills > heliosScoreTimeFromMills) {
            heliosScoreTimeFromMills += 60_000;
            index++;
        }
        heliosScore.getTimeFrom().setTime(heliosScoreTimeFromMills);
        long requestEndTimeMills = request.getEndTime().getTime();
        long heliosScoreTimeToMills = heliosScore.getTimeTo().getTime();
        while (indexDateMills <= requestEndTimeMills && indexDateMills <= heliosScoreTimeToMills && index < scoreIntList.size()) {
            score.getScores().add(scoreIntList.get(index++));
            if (!isDateSetInitial) {
                dateSet.add(new Date(indexDateMills));
            }
            indexDateMills += 60_000;
        }
        isDateSetInitial = true;
        scoreSize = (int) (score.getScores().size() * 1.1);
    }
    response.setDates(new ArrayList<>(dateSet).stream().sorted().map(DateUtils.yyyyMMddHHmm::formatDate).collect(Collectors.toList()));
    return response;
}

This step shaved another ~80 ms, leaving roughly 160 ms total.

Third Optimization

Reduced list property accesses by using subList to bulk‑add elements and avoided per‑iteration list.add calls.

private HeliosGetScoreResponse queryScores(HeliosGetScoreRequest request) {
    // ... same setup as previous version ...
    while (indexDateMills <= requestEndTimeMills && indexDateMills <= heliosScoreTimeToMills && index++ < scoreIntListSize) {
        if (!isDateSetInitial) {
            dateSet.add(new Date(indexDateMills));
        }
        indexDateMills += 60_000;
    }
    score.getScores().addAll(scoreIntList.subList(indexStart, index - 1));
    isDateSetInitial = true;
    scoreSize = (int) (score.getScores().size() * 1.1);
    // ...
}

This optimization contributed another ~100 ms improvement, bringing the latency down to about 60 ms.

Fourth Optimization

Adjusted the SQL query to eliminate an off‑by‑one error that fetched extra rows, and added a fast‑path for single‑record cases, reducing the database round‑trip to 25‑40 ms.

After all optimizations, the total response time for a full‑day query is now well under 100 ms, with the remaining cost primarily in string‑to‑int conversion.

Conclusion

Minimize object creation wherever possible.

SimpleDateFormat incurs significant overhead; prefer caching or alternative date handling.

Even seemingly cheap operations like Date.compareTo , list.size() , and list.add can add up when executed millions of times.

Effective performance tuning requires proper tooling (e.g., Arthas) to pinpoint real bottlenecks rather than relying on intuition.