Analysis of glibc Memory Management and Solutions to an Online Memory Incident

This article, authored by a member of the Vivo Internet Storage team, uses a real online memory incident to illustrate common pitfalls when using glibc for memory management, analyzes the allocation and deallocation mechanisms inside the library, and proposes concrete solutions.

Incident Overview

During an online operation a memory alarm was triggered (memory usage > 4096 KB). Monitoring showed a short‑term spike followed by a rapid drop, yet the process’s resident memory remained high, suggesting a possible memory leak in the user layer. Further investigation revealed that the C runtime library (glibc) might be failing to release memory back to the OS.

glibc Overview

glibc (GNU C Library) provides standard C functions, including a sophisticated memory allocator based on ptmalloc. The article first describes the virtual memory layout of a 32‑bit Linux process (kernel space, stack, mmap region, heap, BSS, data, text).

Heap Operations

The OS supplies brk() and the C runtime supplies sbrk() for expanding the heap:

int brk(void *addr);

void *sbrk(intptr_t increment);

brk moves the program break to allocate or free a contiguous region, while sbrk adjusts the heap size incrementally, reducing fragmentation.

Linux also provides mmap() and munmap() for mapping files or anonymous memory:

void *mmap(void *addr, size_t length, int prot, int flags, int fd, off_t offset);

int munmap(void *addr, size_t length);

ptmalloc Allocator

glibc uses ptmalloc, which organizes memory into arenas, chunks, and bins. An arena (or “allocation zone”) can be the main arena (created by the first thread) or non‑main arenas (created via mmap). The core data structures are:

struct malloc_state {

mutex_t mutex; /* Serialize access */

int flags; /* Flags */

/* ... other fields ... */

mfastbinptr fastbins[NFASTBINS];

mchunkptr top;

mchunkptr last_remainder;

mchunkptr bins[NBINS * 2];

unsigned int binmap[BINMAPSIZE];

struct malloc_state *next;

INTERNAL_SIZE_T system_mem;

INTERNAL_SIZE_T max_system_mem;

};

Chunks are described by:

struct malloc_chunk {

INTERNAL_SIZE_T prev_size; /* Size of previous chunk (if free) */

INTERNAL_SIZE_T size; /* Size including header */

struct malloc_chunk *fd;      /* Forward link (free only) */

struct malloc_chunk *bk;      /* Backward link (free only) */

struct malloc_chunk *fd_nextsize; /* Large bin forward */

struct malloc_chunk *bk_nextsize; /* Large bin backward */

};

The allocator maintains 128 bins, divided into fast bins, unsorted bin, small bins, and large bins. Fast bins handle small, frequently allocated chunks without coalescing; unsorted bin acts as a buffer; small bins manage chunks < 512 B; large bins manage chunks ≥ 512 B.

Allocation Process

Allocation proceeds in three steps: (1) round the request size to a suitable chunk size; (2) search bins (fast → unsorted → small → large) for a fitting chunk; (3) if none is found, extend the top chunk via brk or mmap . The article includes a flow diagram illustrating these steps.

Freeing Process

Freeing a chunk involves: detecting mmaped chunks (use munmap ), coalescing with the top chunk, placing large chunks into unsorted bin (with possible further merging), and inserting small chunks into fast bins. If the top chunk grows beyond a threshold (default 128 KB), the allocator may return memory to the OS.

Fragmentation

The article explains internal fragmentation (wasted space within a chunk) and external fragmentation (free space scattered across bins), showing how the allocator’s merging and splitting strategies mitigate these issues.

Problem Analysis & Solutions

The incident analysis concludes that repeated free/delete only returns memory to the allocator’s bins, not to the OS, especially when freed chunks are not adjacent to the top chunk. Two mitigation strategies are proposed:

Limit the maximum memory usage of the service to avoid excessive heap growth.

Replace glibc’s allocator with jemalloc, which returns idle memory to the OS more aggressively.

Performance Comparison

A benchmark simulating 30 million requests shows that jemalloc reduces peak memory usage and releases memory faster after connections close, achieving roughly a 12 % improvement in memory reclamation speed compared to glibc.

References

glibc Manual

ptmalloc README

Stack Overflow discussion

Zhihu article