Detect and Fix Linux Memory Leaks with memleak and vmstat

Background

Memory leaks occur when a program allocates memory but does not release it, causing the unreclaimed memory to accumulate until the system runs out of RAM. The kernel’s OOM killer may eventually terminate the offending process, but performance degradation typically appears long before that point.

Detecting Memory Leaks with Standard Tools

Start by monitoring the overall memory usage of the system and individual processes:

Use top or ps to watch the resident set size (RSS) of each process. A steadily increasing RSS is a strong indicator of a leak.

When a suspect process is identified, run pmap <PID> to view its memory map and see which regions are growing.

Use vmstat to observe system‑wide memory pressure over time. For example, the command below prints a snapshot every three seconds:

# vmstat 3
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
r  b   swpd   free   buff  cache   si   so   bi   bo   in   cs us sy id wa st
0  0      0 6601824  97620 1098784   0    0    0    0    0    0 62 322  0  0 100  0  0

Key columns:

free : free physical memory (KB)

buff and cache : memory used for buffers and page cache

si/so : swap in/out rates (KB/s)

us/sy/id/wa : CPU time spent in user, system, idle, and I/O wait

The memleak Utility

memleak

is a kernel‑level tracer that records allocation and free events for the whole system or a specific process. It aggregates unfreed allocations and prints the corresponding call stacks, making it possible to pinpoint the exact functions that leak memory.

Common command‑line options -p <PID> – attach to the specified process ID. -a – trace every allocation, even those that are later freed (useful for full‑trace analysis). -o <file> – write the aggregated output to <file> instead of stdout. -s <size> – display only allocations whose size is greater than or equal to <size> bytes. -i <seconds> – set the sampling interval; the tool will emit a summary every <seconds> seconds.

Usage Examples

Real‑time monitoring while saving the output to a log file:

# sudo memleak -p <PID> -o | tee memory_leak.log

Continuous monitoring with watch (updates every 10 seconds, summarising the top 20 stacks):

# watch -n 10 "sudo memleak -p <PID> -c 20"

Sample output (showing the ten stacks with the largest outstanding allocations):

[12:34:56] Top 10 stacks with outstanding allocations:
128 bytes in 4 allocations from stack
    malloc+0x1a [libc]
    create_object+0x25 [myapp]
    process_request+0x67 [myapp]
512 bytes in 2 allocations from stack
    calloc+0x15 [libc]
    init_buffer+0x3e [myapp]

Interpretation:

The timestamp indicates when the snapshot was taken.

Each block lists the total leaked bytes, the number of allocations, and the call stack that allocated them.

By examining the functions in the stack (e.g., create_object, init_buffer), developers can locate the code paths that need fixing or decide to restart the affected service.

Practical Workflow

Identify a suspect process with top / ps (look for a growing RSS).

Confirm the growth with pmap <PID> and vmstat to see system‑wide pressure.

Run memleak using the appropriate options ( -p, -s, -i) to collect allocation data.

Analyze the printed call stacks, fix the leaking code, and optionally restart the service.