Why Your Spring Boot App Consumes 7 GB RAM: Uncovering Native Memory Leaks

Background

After migrating a project to the MDP framework (based on Spring Boot), the system repeatedly reported high Swap usage. Although the JVM was configured with a 4 GB heap, the actual physical memory consumption reached 7 GB, which was abnormal. The JVM parameters used were "-XX:MetaspaceSize=256M -XX:MaxMetaspaceSize=256M -XX:+AlwaysPreTouch -XX:ReservedCodeCacheSize=128m -XX:InitialCodeCacheSize=128m, -Xss512k -Xmx4g -Xms4g,-XX:+UseG1GC -XX:G1HeapRegionSize=4M". The following

shows the memory usage reported by the top command.

Investigation Process

1. Locate memory regions at the Java level

Adding the JVM option -XX:NativeMemoryTracking=detail and restarting the project, the command jcmd <pid> VM.native_memory detail displayed the memory distribution (see

). The committed memory reported by the command was smaller than the physical memory because it includes heap, Code area, and memory allocated via unsafe.allocateMemory or DirectByteBuffer, but excludes native memory allocated by other C code. This suggested that native code was responsible for the excess memory.

Using pmap revealed many 64 MB address ranges that were not listed by jcmd, indicating that these 64 MB blocks were the source of the problem (see

).

2. Locate off‑heap memory at the system level

Since Java‑level tools could not trace the issue, system‑level tools were used.

gperftools

gperftools was employed to monitor memory allocation (see

). The graph showed that memory allocated via malloc peaked at 3 GB and then dropped, stabilising at 700‑800 MB. This raised the question whether native code used malloc or directly used mmap/brk.

strace

Running strace -f -e "brk,mmap,munmap" -p <pid> to trace system calls did not reveal suspicious allocations (see

).

GDB dump

Using GDB to dump suspicious memory regions (identified via /proc/<pid>/smaps) and inspecting the dump with strings revealed JAR‑related information, indicating that the memory was allocated during JAR scanning.

strace during startup

Repeating strace while the application started showed many 64 MB mmap allocations (see

). The corresponding address ranges matched the pmap output (see

).

jstack

Using the thread IDs from strace, jstack <pid> identified the thread stack responsible for the allocations (see

). The analysis pointed to MCC (Meituan Unified Configuration Center) using Reflections to scan JARs via Spring Boot. The underlying Inflater used for JAR decompression allocated off‑heap memory but relied on finalize for release.

Replacing Spring Boot’s ZipInflaterInputStream with the JDK’s implementation eliminated the problem, confirming that the finalizer‑based release was insufficient.

3. Why off‑heap memory was not released

Further investigation showed that the native memory allocator (glibc) kept the freed memory in per‑thread memory pools (64 MB each) instead of returning it to the OS, giving the impression of a leak. The same behaviour was observed with tcmalloc used by gperftools.

To verify, a custom allocator without memory pools was built (see code below) and preloaded via LD_PRELOAD. Monitoring showed that the application consistently requested 700‑800 MB of off‑heap memory, while the OS reported a much larger resident set due to the allocator’s pooling strategy.

#include<sys/mman.h>
#include<stdlib.h>
#include<string.h>
#include<stdio.h>
// 64‑bit machine, sizeof(size_t) == sizeof(long)
void* malloc ( size_t size )
{
   long* ptr = mmap( 0, size + sizeof(long), PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANONYMOUS, 0, 0 );
   if (ptr == MAP_FAILED) {
   	return NULL;
   }
   *ptr = size;                     // First 8 bytes contain length.
   return (void*)(&ptr[1]);        // Memory after length variable
}

void *calloc(size_t n, size_t size) {
   void* ptr = malloc(n * size);
   if (ptr == NULL) {
   	return NULL;
   }
   memset(ptr, 0, n * size);
   return ptr;
}

void *realloc(void *ptr, size_t size)
{
   if (size == 0) {
   	free(ptr);
   	return NULL;
   }
   if (ptr == NULL) {
   	return malloc(size);
   }
   long *plen = (long*)ptr;
   plen--;                          // Reach top of memory
   long len = *plen;
   if (size <= len) {
   	return ptr;
   }
   void* rptr = malloc(size);
   if (rptr == NULL) {
   	free(ptr);
   	return NULL;
   }
   rptr = memcpy(rptr, ptr, len);
   free(ptr);
   return rptr;
}

void free (void* ptr )
{
   if (ptr == NULL) {
   	return;
   }
   long *plen = (long*)ptr;
   plen--;                          // Reach top of memory
   long len = *plen;               // Read length
   munmap((void*)plen, len + sizeof(long));
}

Summary

The root cause was that MCC’s default configuration scanned all JARs, causing Spring Boot’s ZipInflaterInputStream to allocate off‑heap memory without explicit release. GC eventually freed the memory via finalize, but glibc retained the pages in its per‑thread memory pool, making it appear as a leak. Restricting MCC’s scan path solved the issue, and newer Spring Boot versions (2.0.5.RELEASE) now proactively release the off‑heap memory, eliminating the problem.