How We Traced and Fixed Excessive Native Memory Usage After Migrating to Spring Boot

Background

After migrating a project to the MDP framework built on Spring Boot, the system started reporting high swap usage. Although the JVM was configured with a 4 GB heap, physical memory consumption reached 7 GB, indicating abnormal off‑heap memory usage.

JVM Configuration

-XX:MetaspaceSize=256M
-XX:MaxMetaspaceSize=256M
-XX:+AlwaysPreTouch
-XX:ReservedCodeCacheSize=128m
-XX:InitialCodeCacheSize=128m
-Xss512k -Xmx4g -Xms4g -XX:+UseG1GC
-XX:G1HeapRegionSize=4M

Investigation Process

1. Locate memory regions with Java‑level tools

Add -XX:NativeMemoryTracking=detail to the JVM arguments, restart, and run jcmd <pid> VM.native_memory detail. The output showed that the committed memory reported by jcmd was smaller than the physical usage, suggesting native code allocations were not accounted for.

Using pmap revealed many 64 MB address ranges that were absent from the jcmd report, pointing to native allocations.

2. Use system‑level tools to trace off‑heap memory

First tried gperftools to monitor native allocations. The profiler showed a peak of 3 GB malloced memory that later settled around 700‑800 MB.

Because gperftools did not capture the suspect allocations, strace -f -e "brk,mmap,munmap" -p <pid> was run. The trace did not reveal obvious allocation calls.

Next, gdb was used to dump suspicious memory regions ( gdp -pid <pid> followed by dump memory mem.bin startAddress endAddress) and inspect them with strings. The dump contained JAR metadata, indicating that the memory was allocated during JAR unpacking.

Running strace at application start captured many 64 MB mmap calls, confirming that the memory was allocated early in the boot process.

3. Identify the root cause in Spring Boot

Investigation revealed that the Meituan Configuration Center (MCC) uses Reflections to scan all JARs. During scanning, Spring Boot’s ZipInflaterInputStream wraps Inflater to decompress JAR entries, allocating off‑heap memory that was only released in the finalize method of Inflater.

Because finalization depends on GC, the native memory remained allocated for the lifetime of the scan, causing the observed swap pressure. Modifying MCC’s configuration to limit the scan to specific packages eliminated the excessive allocation.

Further analysis showed that the native allocator (glibc) retains freed pages in per‑thread memory arenas (64 MB each). Even after GC freed the Java‑level memory, the OS kept the pages, making it appear as a leak.

Additional Experiments

A custom allocator without arena pooling was built (compiled with gcc zjbmalloc.c -fPIC -shared -o zjbmalloc.so) and preloaded via LD_PRELOAD. Tests showed that native memory usage stayed around 700‑800 MB regardless of the allocator, while the process’s RSS varied due to page‑level rounding and lazy allocation.

Conclusion

The excessive off‑heap memory was caused by Spring Boot’s default JAR scanning implementation, which relied on Inflater without explicit deallocation. Updating the scan path or upgrading to Spring Boot 2.0.5 (which releases the native buffer proactively) resolves the issue. Additionally, glibc’s arena‑based allocator can retain freed memory, making it appear as a leak at the OS level.