How a Hidden Memory Leak Stole Hours: Debugging Multithreaded C++ Code

During a routine performance check the author observed a sudden spike in memory usage of a service and suspected a memory leak.

Initial Investigation

Two recent code commits were identified; one belonged to the author and the other only changed configuration. After confirming the configuration change could not cause a leak, the author rolled back his own changes and began debugging.

What Is a Memory Leak?

A memory leak occurs when allocated memory is not released back to the operating system. In C++ this typically looks like allocating an object with new (or malloc) and never calling delete (or free).

obj* o = new obj();
// ...
// missing delete o;

Refactoring and Parallelization

The author was parallelizing a previously single‑threaded section of code, splitting it into two threads. This introduced shared data that became the source of the leak.

Memory allocated with new/delete Memory allocated from a custom memory pool

All allocations appeared to be correctly freed, yet the leak persisted, prompting the use of a memory‑checking tool.

Choosing a Detection Tool

Common tools include Valgrind (no recompilation needed but slows execution 20‑30×) and gperftools (requires recompilation). The author opted for Valgrind due to its ease of use.

Root Cause: Shared Data Race

The problematic code allocated a protobuf object via a global pointer:

auto* pb = global->mutable_obj();

When accessed from both threads, the following race occurs:

Thread A checks global->obj is NULL and is about to allocate.

Thread switch occurs before allocation.

Thread B performs the same check, allocates, then switches.

Thread A resumes and allocates again, leaving the first allocation unreachable.

This double allocation without a corresponding delete results in a classic memory leak.

Fix

The fix was to eliminate the shared pointer by introducing a thread‑local variable, ensuring each thread works on its own instance. The issue was resolved in about four hours.

Lessons Learned

Avoid shared mutable state when parallelizing code; prefer thread‑local data.

When debugging memory leaks, prioritize locating shared data accessed by multiple threads.

Tools like Valgrind can quickly pinpoint invalid writes and definite leaks.

Valgrind Example

Sample program with an out‑of‑bounds write and a leak:

#include <stdlib.h>
void f(void) {
    int* x = malloc(10 * sizeof(int));
    x[10] = 0; // out‑of‑bounds
}
int main() {
    f();
    return 0;
}

Running valgrind --leak-check=yes myprog produces output indicating an invalid write and a “definitely lost” leak, helping the developer locate the exact source lines.

Conclusion

Writing correct multithreaded code is challenging; the primary source of bugs is shared resources. Before using locks, ask whether sharing is necessary. If a leak occurs, focus first on shared data accessed by multiple threads.