Uncovering Netty’s Hidden Memory Leak: A Deep Dive into the Recycler Object Pool

Introduction

The author discovered a deep‑seated memory‑leak bug while reviewing Netty 4.1.56.Final source code, specifically in the Recycler implementation used for object pooling.

Object‑Pool Overview

Netty’s Recycler provides a lock‑free pool where each thread has its own Stack (an array‑based stack) to store recycled objects, avoiding synchronization overhead during allocation.

Key Concepts

Stack : Per‑thread storage of recycled objects; contains an array elements[] and a linked list of WeakOrderQueue nodes for cross‑thread recycling.

WeakOrderQueue : Holds objects recycled by threads other than the creator; implemented as a linked list of Link nodes, each storing up to LINK_CAPACITY (default 16) objects.

DefaultHandle : Wrapper for pooled objects; tracks recycleId, lastRecycledId, and a reference to its owning Stack.

Lock‑Free Allocation

When Recycler.get() is called, the current thread retrieves its Stack via a FastThreadLocal. If the stack’s array is non‑empty, an object is popped; otherwise the stack attempts to scavenge objects from its WeakOrderQueue chain. If scavenging fails, a new object is created via the abstract newObject() method.

DefaultHandle<T> pop() { ... }

Cross‑Thread Recycling

If a thread other than the creator recycles an object, the DefaultHandle.recycle() method forwards the object to Stack.push(). push() checks whether the current thread matches the stack’s creator thread. If not, it uses a WeakHashMap<Stack,WeakOrderQueue> stored in a thread‑local DELAYED_RECYCLED to locate or create a WeakOrderQueue for the creator’s stack.

void pushLater(DefaultHandle<?> item, Thread thread) { ... }

The map is weak‑referenced so that if the creator thread dies, its entry can be garbage‑collected, preventing leaks.

Bug Discovery

The author noticed that when the creator thread had already terminated ( threadRef.get() == null), recycled objects were still added to the creator’s WeakOrderQueue. This created a reference chain pooledObject → DefaultHandle → Stack → Thread, preventing the dead thread and its stack from being reclaimed, leading to a memory leak.

Proposed Fix (PR 11865)

The fix adds a guard in Stack.push() to ignore recycling when threadRef.get() == null and clears the handle’s stack reference to break the chain.

if (threadRef.get() == null) { item.stack = null; return; }

Follow‑Up and Refactor

Netty later refactored the entire Recycler in version 4.1.71.Final, introducing a LocalPool that replaces Stack. The same leak persisted until it was finally fixed in 4.1.74.Final (PR 11996). The refactor also moved the cleanup of per‑thread pools to FastThreadLocal.onRemoval(), clearing references when a thread exits.

Key Takeaways

Lock‑free designs must still consider thread‑lifecycle edge cases to avoid hidden leaks.

Weak references and WeakHashMap are essential for allowing dead threads to be reclaimed.

Controlling recycling frequency (e.g., 1/8 of objects) prevents uncontrolled pool growth.

Even well‑tested libraries can harbor subtle bugs that surface only under specific concurrency patterns.