How Netty Detects Memory Leaks: Deep Dive into LeakAwareBuffer and ResourceLeakDetector

Based on Netty 4.1.112.Final.

This is the final article in the Netty memory‑management series. The first article introduced the ByteBuf design using UnpooledByteBuf, and the second article dissected Netty's memory pool. Here we focus on how Netty detects memory leaks for both pooled and unpooled ByteBufs.

1. Memory Leak Detection Design Principles

Both unpooled ( UnpooledByteBuf) and pooled ( PooledByteBuf) allocations are finally wrapped in a LeakAwareBuffer before being returned to the user.

public final class UnpooledByteBufAllocator {
    @Override
    protected ByteBuf newDirectBuffer(int initialCapacity, int maxCapacity) {
        final ByteBuf buf;
        if (PlatformDependent.hasUnsafe()) {
            buf = noCleaner ? new InstrumentedUnpooledUnsafeNoCleanerDirectByteBuf(this, initialCapacity, maxCapacity)
                           : new InstrumentedUnpooledUnsafeDirectByteBuf(this, initialCapacity, maxCapacity);
        } else {
            buf = new InstrumentedUnpooledDirectByteBuf(this, initialCapacity, maxCapacity);
        }
        // If leak detection is enabled, wrap the buffer.
        return disableLeakDetector ? buf : toLeakAwareBuffer(buf);
    }
}

public class PooledByteBufAllocator {
    private final PoolThreadLocalCache threadCache;
    @Override
    protected ByteBuf newDirectBuffer(int initialCapacity, int maxCapacity) {
        PoolThreadCache cache = threadCache.get();
        PoolArena<ByteBuffer> directArena = cache.directArena;
        final ByteBuf buf;
        if (directArena != null) {
            buf = directArena.allocate(cache, initialCapacity, maxCapacity);
        } else {
            buf = PlatformDependent.hasUnsafe() ?
                  UnsafeByteBufUtil.newUnsafeDirectByteBuf(this, initialCapacity, maxCapacity) :
                  new UnpooledDirectByteBuf(this, initialCapacity, maxCapacity);
        }
        // Wrap with LeakAwareByteBuf if leak detection is enabled.
        return toLeakAwareBuffer(buf);
    }
}

Netty prefers manual release of DirectByteBuf to avoid the delayed native‑memory reclamation caused by the JVM GC on java.nio.DirectByteBuffer. However, manual release can lead to leaks if release() is forgotten, so Netty introduces LeakAwareBuffer to detect such cases.

2. Netty's Memory Leak Detection Mechanism

Leak detection is triggered only when five conditions are met:

The application enables leak detection.

The ByteBuf must be garbage‑collected before a leak can be detected.

Detection occurs on the next memory allocation after GC.

Detection is sampled; only buffers that pass the sampling interval are examined.

The logger level must be ERROR because leak reports are logged at ERROR level.

Netty defines four detection levels:

public enum Level {
    DISABLED,
    SIMPLE,
    ADVANCED,
    PARANOID;
}

The level can be set via the JVM option -Dio.netty.leakDetection.level. DISABLED turns off detection. SIMPLE (default) samples every SAMPLING_INTERVAL buffers (default 128) and records only the allocation stack. ADVANCED records allocation and recent access stacks, while PARANOID examines every buffer without sampling.

Sampling is performed with:

PlatformDependent.threadLocalRandom().nextInt(samplingInterval) == 0

If the random number is zero, the buffer is tracked; otherwise it is ignored.

3. Memory Leak Detection Design Models

3.1 ResourceLeakDetector

ResourceLeakDetector

holds the global state for leak detection: a set allLeaks of active DefaultResourceLeak objects and a ReferenceQueue that receives notifications when a tracked ByteBuf is reclaimed.

public class ResourceLeakDetector<T> {
    private static Level level;
    private final Set<DefaultResourceLeak<?>> allLeaks =
        Collections.newSetFromMap(new ConcurrentHashMap<DefaultResourceLeak<?>, Boolean>());
    private final ReferenceQueue<Object> refQueue = new ReferenceQueue<>();
    private final String resourceType;
    private final int samplingInterval;
    private volatile LeakListener leakListener;
    // ... core methods track() and reportLeak() ...
}

When a buffer is allocated, track() decides whether to create a DefaultResourceLeak based on the current level and sampling interval. If created, the leak object is added to allLeaks. When the buffer is GC‑ed, the JVM moves the weak reference to refQueue. The detector then checks whether the leak object is still in allLeaks; if it is, a leak is reported.

3.2 ResourceLeakTracker

The interface ResourceLeakTracker is implemented by DefaultResourceLeak. It records access stacks (as TraceRecord objects) and provides record() methods that can optionally include a custom hint.

final class DefaultResourceLeak<T> extends WeakReference<Object>
        implements ResourceLeakTracker<T>, ResourceLeak {
    private volatile TraceRecord head;
    private volatile int droppedRecords;
    private final Set<DefaultResourceLeak<?>> allLeaks;
    private final int trackedHash;
    // Constructor registers the leak and creates the first TraceRecord.
    DefaultResourceLeak(Object referent, ReferenceQueue<Object> refQueue,
                       Set<DefaultResourceLeak<?>> allLeaks, Object initialHint) {
        super(referent, refQueue);
        this.trackedHash = System.identityHashCode(referent);
        allLeaks.add(this);
        head = initialHint == null ? new TraceRecord(TraceRecord.BOTTOM)
                                 : new TraceRecord(TraceRecord.BOTTOM, initialHint);
        this.allLeaks = allLeaks;
    }
    @Override
    public void record() { record0(null); }
    @Override
    public void record(Object hint) { record0(hint); }
    // ... stack management and report generation ...
}

The stack holds the most recent access record at the top and the allocation record at the bottom. The number of records is limited by -Dio.netty.leakDetection.targetRecords (default 4). When the limit is exceeded, the top record is probabilistically dropped.

3.3 TraceRecord

Each TraceRecord extends Throwable so that its stack trace can be captured automatically. The toString() method formats the stack, optionally prefixing a custom hint.

private static final class TraceRecord extends Throwable {
    private static final TraceRecord BOTTOM = new TraceRecord() {
        @Override public Throwable fillInStackTrace() { return this; }
    };
    private final String hintString;
    private final TraceRecord next;
    private final int pos;
    TraceRecord(TraceRecord next) { this(next, null); }
    TraceRecord(TraceRecord next, Object hint) {
        this.next = next;
        this.pos = next.pos + 1;
        this.hintString = hint == null ? null : hint.toString();
    }
    @Override public String toString() {
        StringBuilder sb = new StringBuilder(2048);
        if (hintString != null) sb.append("\tHint: ").append(hintString).append('
');
        for (StackTraceElement e : getStackTrace()) {
            // Skip internal frames and format the rest.
            sb.append('\t').append(e).append('
');
        }
        return sb.toString();
    }
}

3.4 LeakAwareByteBuf

When a buffer is tracked, Netty wraps it in a LeakAwareByteBuf. The wrapper type depends on the detection level: SIMPLE → SimpleLeakAwareByteBuf (records only allocation). ADVANCED or PARANOID → AdvancedLeakAwareByteBuf (records allocation and selected accesses).

protected static ByteBuf toLeakAwareBuffer(ByteBuf buf) {
    ResourceLeakTracker<ByteBuf> leak;
    switch (ResourceLeakDetector.getLevel()) {
        case SIMPLE:
            leak = AbstractByteBuf.leakDetector.track(buf);
            if (leak != null) buf = new SimpleLeakAwareByteBuf(buf, leak);
            break;
        case ADVANCED:
        case PARANOID:
            leak = AbstractByteBuf.leakDetector.track(buf);
            if (leak != null) buf = new AdvancedLeakAwareByteBuf(buf, leak);
            break;
        default:
            break;
    }
    return buf;
}

SimpleLeakAwareByteBuf

forwards all operations to the underlying buffer and closes the leak when the reference count reaches zero:

public class SimpleLeakAwareByteBuf extends WrappedByteBuf {
    private final ByteBuf trackedByteBuf;
    private final ResourceLeakTracker<ByteBuf> leak;
    @Override public boolean release() {
        if (super.release()) {
            closeLeak();
            return true;
        }
        return false;
    }
    private void closeLeak() { leak.close(trackedByteBuf); }
}

AdvancedLeakAwareByteBuf

adds a call to leak.record() (or record(hint)) on most read/write operations, unless the JVM option -Dio.netty.leakDetection.acquireAndReleaseOnly is set to true, in which case only retain(), release() and touch() are recorded.

public class AdvancedLeakAwareByteBuf extends SimpleLeakAwareByteBuf {
    @Override public byte readByte() {
        recordLeakNonRefCountingOperation(leak);
        return super.readByte();
    }
    @Override public ByteBuf writeByte(int value) {
        recordLeakNonRefCountingOperation(leak);
        return super.writeByte(value);
    }
    static void recordLeakNonRefCountingOperation(ResourceLeakTracker<ByteBuf> leak) {
        if (!ACQUIRE_AND_RELEASE_ONLY) {
            leak.record();
        }
    }
}

When a tracked buffer is properly released, its DefaultResourceLeak is removed from allLeaks, its weak reference is cleared, and its stack is discarded, preventing false positives.

Summary

To trigger Netty's leak detection, the following must be true:

Leak detection is enabled via the JVM option.

The ByteBuf has been garbage‑collected.

A subsequent memory allocation occurs.

The buffer passes the sampling interval (unless PARANOID is used).

The logger level includes ERROR.

The detection level can be set with -Dio.netty.leakDetection.level: DISABLED – no detection. SIMPLE – sampled detection; reports only allocation stack; sampling interval configurable via -Dio.netty.leakDetection.samplingInterval. ADVANCED – sampled detection with recent access stacks; number of recorded accesses limited by -Dio.netty.leakDetection.targetRecords. PARANOID – full detection of every buffer; most detailed reports.

For production environments it is recommended to keep detection disabled or at SIMPLE level to avoid the memory overhead of storing stack traces (each TraceRecord consumes ~2 KB). Detailed detection is useful in testing when tracking down elusive leaks.