Root Cause Analysis of Excessive Swap Usage in a Spring Boot Project: Native Memory Leak Triggered by MCC Package Scanning

Background

After migrating a project to the MDP framework (based on Spring Boot), the system repeatedly reported high swap usage. Although the JVM was configured with -Xmx4g -Xms4g , the physical memory usage reached 7 GB, which was abnormal.

The JVM parameters were:

-XX:MetaspaceSize=256M -XX:MaxMetaspaceSize=256M -XX:+AlwaysPreTouch -XX:ReservedCodeCacheSize=128m -XX:InitialCodeCacheSize=128m, -Xss512k -Xmx4g -Xms4g,-XX:+UseG1GC -XX:G1HeapRegionSize=4M

Investigation Process

1. Locate memory regions from the Java side

Added -XX:NativeMemoryTracking=detailJVM and restarted the project. Using jcmd pid VM.native_memory detail showed the memory distribution, but the committed memory was smaller than the physical usage because it did not include native allocations made via unsafe.allocateMemory or DirectByteBuffer .

Using pmap revealed many 64 MB address ranges that were not reported by jcmd , suggesting native code allocations.

2. Locate off‑heap memory from the system side

Since Java tools could not pinpoint the source, system‑level tools were employed.

gperftools was used to monitor malloc activity. The graph showed memory spiking to 3 GB and then stabilising around 700‑800 MB.

Next, strace -f -e "brk,mmap,munmap" -p pid was run, but no suspicious allocations were observed.

Because strace missed the allocations, gdb was used to dump memory regions ( gdp -pid pid followed by dump memory mem.bin startAddress endAddress ) and the dump was inspected with strings mem.bin . The output indicated JAR‑file loading during startup.

Repeating strace at application start captured many 64 MB mmap calls.

The corresponding address ranges were verified with pmap .

Using the thread IDs from strace , jstack pid identified the threads responsible for the allocations.

The investigation revealed that MCC (Meituan Configuration Center) uses Reflections to scan all JARs. During this scan, Spring Boot’s InflaterInputStream (wrapped as ZipInflaterInputStream ) allocates off‑heap memory via native malloc but does not release it promptly.

Replacing the Spring Boot inflater with the JDK’s built‑in implementation eliminated the memory growth.

3. Why the off‑heap memory was not released

Although Inflater implements finalize() to free native memory, the GC relied on finalization, and the underlying glibc memory allocator kept the pages in its arena (64 MB per thread) instead of returning them to the OS, giving the impression of a leak.

A custom malloc implementation without arenas was built and preloaded via LD_PRELOAD to verify the hypothesis. The custom allocator used mmap directly:

#include <sys/mman.h>
#include <stdlib.h>
#include <string.h>
#include <stdio.h>

void* malloc(size_t size) {
    long* ptr = mmap(0, size + sizeof(long), PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, 0, 0);
    if (ptr == MAP_FAILED) return NULL;
    *ptr = size;
    return (void*)(&ptr[1]);
}

void* calloc(size_t n, size_t size) {
    void* ptr = malloc(n*size);
    if (!ptr) return NULL;
    memset(ptr, 0, n*size);
    return ptr;
}

void* realloc(void* ptr, size_t size) {
    if (size == 0) { free(ptr); return NULL; }
    if (!ptr) return malloc(size);
    long* plen = (long*)ptr; plen--; long len = *plen;
    if (size <= len) return ptr;
    void* rptr = malloc(size);
    if (!rptr) { free(ptr); return NULL; }
    memcpy(rptr, ptr, len);
    free(ptr);
    return rptr;
}

void free(void* ptr) {
    if (!ptr) return;
    long* plen = (long*)ptr; plen--; long len = *plen;
    munmap((void*)plen, len + sizeof(long));
}

Tests showed that even though the custom allocator kept only 800 MB of off‑heap memory, the OS reported about 1.7 GB of resident memory because mmap reserves whole pages and the kernel allocates physical pages lazily.

Conclusion

The excessive swap usage was caused by MCC’s default package‑scanning configuration, which triggered Spring Boot’s ZipInflaterInputStream to allocate large off‑heap buffers that were only freed by GC finalization. The glibc arena kept the memory from being returned to the OS, appearing as a leak. Limiting the scan to specific JARs or upgrading to Spring Boot 2.0.5 (which releases the inflater’s native memory proactively) resolves the problem.