Dynamic‑Tracing Based Memory‑Leak (Growth) Analysis for MySQL‑Proxy in TDSQL

Memory leaks are a common pain point for developers. Traditional analysis tools such as gdb and Valgrind are often inefficient for locating leaks in production environments. This article, authored by Tencent backend engineer Xing Mengbang, uses a real‑world memory‑leak case in TDSQL’s mysql‑proxy to demonstrate a generic analysis method based on dynamic tracing.

Background : In a private TDSQL deployment, mysql‑proxy’s memory usage continuously grew during heavy request forwarding, eventually causing OOM and affecting user services. Conventional tools (gdb, Valgrind) proved slow or unsuitable for the private‑cloud scenario.

Basic Concepts : Application‑level memory leaks can be divided into kernel memory leaks (detectable with kmemleak) and user‑space leaks, which include heap leaks and memory‑mapping leaks. The article focuses on the latter.

Memory Allocation Process (illustrated in the original figures): 1. Application calls malloc / free (or variants) provided by the allocator (e.g., libc, jemalloc). 2. The allocator may expand the heap via the brk system call for small allocations. 3. For large allocations it uses mmap to reserve virtual memory. 4. The first write to a newly‑mapped page triggers a page‑fault, causing the OS to allocate a physical page and update the page table.

Limitations of Traditional Tools : gdb interferes with normal program execution and requires source‑level knowledge. Valgrind Memcheck needs to restart the program, slows execution by 20‑30×, and does not support jemalloc (used by mysql‑proxy).

Dynamic‑Tracing Based Generic Analysis : Tracks allocator‑related functions and page‑fault events as discrete events. Aggregates un‑released allocations and page‑fault stacks to produce flame‑graphs, allowing developers to focus on a few suspect code paths.

Two Analysis Methods :

1) Memory‑Allocator Behaviour Analysis

Developed a tool memstacks that generates two types of flame‑graphs:

Full allocation flame‑graph (tracks malloc‑family calls without subtracting frees).

Un‑released allocation flame‑graph (tracks malloc‑family and free, showing only memory still allocated).

Example commands:

# Step 1: Trace for 60 s and output collapsed stacks
./memstacks -p $(pgrep -nx mysql-proxy) -af 60 > all_mallocs.stacks

# Step 2: Generate flame‑graph
./flamegraph.pl --color=mem --title="All malloc() bytes Flame Graph" \
    --countname="bytes" < all_mallocs.stacks > all_mallocs.svg

For un‑released allocations:

# Step 1: Trace for 60 s, record only un‑freed allocations
memstacks -p $(pgrep -nx mysql-proxy) -f 60 > unfreed_mallocs.stacks

# Step 2: Generate flame‑graph
./flamegraph.pl --color=mem --title="Unfreed malloc() bytes Flame Graph" \
    --countname="bytes" < unfreed_mallocs.stacks > unfreed_mallocs.svg

The un‑released flame‑graph revealed ~27.75 MiB of leaked memory, matching the RSS increase observed by pidstat . The primary leak originated from tdsql::Item_param::set_str .

2) Page‑Fault Event Analysis

Instead of tracking virtual allocations, this method focuses on the kernel‑side page‑fault events that actually increase RSS. A new tool pgfaultstacks was built to filter out page‑faults that map already‑present pages and to exclude pages that were later freed.

Example command using the existing tool perf (for reference):

perf record -p $(pgrep -nx mysql-proxy) -e page-faults -c 1 -g -- sleep 60 > pgfault.stacks

Custom tool usage:

# Step 1: Trace page‑faults for 60 s
pgfaultstacks -p $(pgrep -nx mysql-proxy) -f 60 > pgfault.stacks

# Step 2: Generate flame‑graph
./flamegraph.pl --color=mem --title="Page Fault Flame Graph" \
    --countname="pages" < pgfault.stacks > pgfault.svg

The resulting flame‑graph showed 17 801 newly allocated pages (≈ 60 MiB RSS increase) with the most significant stack involving g_string_append_printf . Compared with the generic version, the data volume was reduced by more than 20×, making the analysis much more focused.

Conclusion : By applying dynamic tracing to both allocator behaviour and page‑fault events, developers can quickly narrow down the code paths that cause memory growth. The custom tools memstacks and pgfaultstacks enable efficient, production‑friendly memory‑leak diagnostics without heavy instrumentation or source‑level debugging.