Analysis and Resolution of Frequent Full GC in an Online Service

In early April an alert indicated that one instance of an online Java service was experiencing extremely frequent full garbage collections (Full GC), causing request timeouts and downstream service failures.

The immediate recovery step was to take the affected instance offline after dumping its heap for later analysis.

Investigation revealed that the root cause was a massive data load triggered by a rarely used request; the service loaded a large amount of data from the database, which filled the heap and forced Full GC.

Two main remediation paths were pursued: adjusting JVM parameters to reduce object promotion and Full GC frequency, and modifying the application code to prevent large‑scale data queries.

JVM parameter adjustments aimed to minimize the number of short‑lived objects moving to the old generation. The following options were applied:

-Xmx5g
-Xms5g
-XX:MaxMetaspaceSize=512M
-XX:MaxTenuringThreshold=15
-XX:MetaspaceSize=512M
-XX:NewSize=2560M
-XX:MaxNewSize=2560M
-XX:SurvivorRatio=8
-XX:+UseConcMarkSweepGC
-XX:+PrintGCApplicationStoppedTime
-XX:+UseCMSCompactAtFullCollection
-XX:CMSInitiatingOccupancyFraction=85
-Xloggc:/opt/zcy/modules/agreement-center/gc.log
-XX:CMSFullGCsBeforeCompaction=2
-XX:+CMSScavengeBeforeRemark
-XX:+UseCMSInitiatingOccupancyOnly

Key changes included enlarging the young generation to half of the heap, raising the tenuring threshold to keep objects in the young generation longer, and adjusting the survivor‑to‑eden ratio.

Code‑logic adjustments introduced a MyBatis interceptor that limits the size of query results. When a query returns more rows than a configured threshold, a warning is logged and an alert can be raised. The interceptor implementation is shown below:

@Intercepts(@Signature(type = ResultSetHandler.class, method = "handleResultSets", args = {Statement.class}))
@Slf4j
public class QueryDataSizeInterceptor implements Interceptor {
    private Integer querySizeLimit;
    private Boolean isOpen;
    public QueryDataSizeInterceptor(Integer querySizeLimit, Boolean isOpen) {
        this.querySizeLimit = querySizeLimit;
        this.isOpen = isOpen;
    }
    @Override
    public Object intercept(Invocation invocation) throws Throwable {
        try {
            if (isOpen) {
                processIntercept(invocation.getArgs());
            }
        } catch (Throwable throwable) {
            log.warn("QueryDataSizeInterceptor.failed,cause:{}", Throwables.getStackTraceAsString(throwable));
        }
        return invocation.proceed();
    }
    private void processIntercept(final Object[] queryArgs) {
        Statement statement = (Statement) queryArgs[0];
        try {
            HikariProxyResultSet resultSet = (HikariProxyResultSet) statement.getResultSet();
            MetaObject metaObject1 = SystemMetaObject.forObject(resultSet);
            RowDataStatic rs = (RowDataStatic) metaObject1.getValue("delegate.rowData");
            if (Objects.nonNull(rs) && !rs.wasEmpty() && rs.size() >= querySizeLimit) {
                MetaObject metaObject2 = SystemMetaObject.forObject(statement);
                String sql = (String) metaObject2.getValue("delegate.originalSql");
                log.warn("current.query.size.is.too.large,size:{},sql:{}", rs.size(), sql);
            }
        } catch (Throwable throwable) {
            log.warn("QueryDataSizeInterceptor.failed,cause:{}", Throwables.getStackTraceAsString(throwable));
        }
    }
    @Override
    public Object plugin(Object target) {
        return Plugin.wrap(target, this);
    }
    @Override
    public void setProperties(Properties properties) {}
}

Later the same issue resurfaced. With the interceptor logging enabled, the trace ID pointed to a query that fetched over 300,000 rows from the ag_protocol table, confirming that the large result set caused the Full GC.

Long total = protocolQualificationManager.count(criteria);
if (total == 0) {
    return Response.ok(new Paging<>(0L, Collections.EMPTY_LIST));
}
List<AgProtocolQualification> result = protocolQualificationManager.paging(criteria);
Set<Long> protocolIds = FluentIterable.from(result)
    .transform(k -> k.getProtocolId())
    .toSet();
List<AgProtocol> protocols = agProtocolDao.queryByIds(Lists.newArrayList(protocolIds));

The corresponding MyBatis mapper lacked a safeguard for empty ids parameters, resulting in a full‑table query:

<select id="queryByIds" parameterType="java.util.List" resultMap="defaultResultMap">
SELECT
<include refid="allColumns"/>
FROM ag_protocol
<where>
    <if test="ids != null and ids.size != 0">
        and id in
        <foreach collection="ids" open="(" close=")" separator="," item="id">
            #{id}
        </foreach>
    </if>
    <!-- added safeguard to prevent full‑table scan -->
    <if test="ids == null or ids.size == 0">
        and false
    </if>
    <include refid="not_deleted"/>
</where>
</select>

Key lessons and pitfalls highlighted include: never allow dynamic SQL to fall back to a full‑table query when all conditions are missing; enforce limits (e.g., LIMIT) at the application or database level; redesign data structures to avoid massive result sets; and add protective checks in mappers to prevent accidental full scans.