Skip to content
Go 1.27 Goroutine Leak Profiler: Detecting Permanently Blocked Goroutines in Production

Go 1.27 Goroutine Leak Profiler: Detecting Permanently Blocked Goroutines in Production

Go 1.27 introduces a goroutine leak profiler that uses a modified garbage collector to identify permanently blocked goroutines on channels and sync primitives in running programs, including production, with minimal overhead and near-zero false positives, demonstrated through real-world examples from CockroachDB, etcd, Kubernetes, and Moby.

Go Programming World
Go Programming World
Go Programming World

Go's concurrency features are powerful but can lead to goroutine leaks when goroutines block forever on conditions that never become true. Existing tools like the race detector, goleak, and synctest help in tests but cannot inspect production systems. Go 1.27 adds a goroutine leak profiler that precisely finds leaked goroutines blocked on channels or sync primitives in live programs, including production, with almost no false positives.

Example: Concurrent Workers

A function processWorkItems spawns a worker per item, each sending a result on an unbuffered channel ch. If an error occurs, the function returns early, leaving the remaining workers blocked on ch <- result{res, err}. This pattern appears in real services, including Uber's.

Debugging with the Profiler

The profile is available via runtime/pprof (type goroutineleak) or automatically at /debug/pprof/goroutineleak when net/http/pprof is imported. A complete runnable example is provided. After building and running, collect the profile:

$ curl http://localhost:6060/debug/pprof/goroutineleak > leak.prof
$ go tool pprof leak.prof
Type: goroutineleak
Time: 2026-03-01 13:19:49 UTC
Entering interactive mode (type "help" for commands, "o" for options)
(pprof) list processWorkItems
Total: 116
ROUTINE ======================== main.processWorkItems.func1 in .../main.go
         0        116 (flat, cum) 100% of Total
         .          .     31:         go func() {
         .          .     32:                 res, err := processWorkItem(w)
         .        116     33:                 ch <- result{res, err}
         .          .     34:         }()

The profile points directly to line 33 where the send blocks. The longer the program runs, the more goroutines leak.

Fixing the Leak

Adding a buffer to the channel equal to the number of workers ( make(chan result, len(ws))) lets all sends complete even if the receiver stops early.

Implementation Internals

The profiler reuses Go's concurrent tri-color mark-sweep GC (Green Tea variant) with five key changes:

Only unblocked goroutines are treated as GC roots (mark roots).

The marking phase follows references from those roots, implicitly marking only memory reachable from live goroutines.

After marking, any blocked goroutine that is blocked on a marked concurrency primitive is added to the root set, and marking continues (inductive step).

Goroutines never added to the root set are marked as leaked.

A final marking pass runs with leaked goroutines also as roots so the GC completes normally.

Two diagrams from the Go blog illustrate the difference between standard GC reachability and the leak profiler's liveness analysis.

Standard GC reachability
Standard GC reachability
Leak profiler liveness analysis
Leak profiler liveness analysis

Limitations

Memory overreach : If a primitive is reachable from a global variable or runnable goroutine, blocked goroutines on it are never reported, even if the primitive will never be used again.

Non-standard blocking : Only built-in primitives are covered: channel send/receive (including nil channels), blocking select (no default), and sync.Mutex, sync.RWMutex, sync.WaitGroup, sync.Cond. File I/O, network I/O, syscalls, and custom synchronization (e.g., spinlocks) are excluded.

Non-determinism : Leaks are only detectable after they occur. Best results come from combining periodic production profiling with goleak and synctest in tests.

Performance Impact

Memory overhead is negligible. CPU overhead can be higher than a regular GC in pathological "daisy-chain" cases where a runnable goroutine G₀ holds primitive P₁ blocking G₁, which holds P₂ blocking G₂, etc. This serializes marking and can make a GC cycle O(n²) in the number of goroutines. The first limitation is inherent; the second may be optimized. Periodic collection (e.g., every 4 hours) keeps overhead low while retaining detection capability.

Daisy-chain worst-case scenario
Daisy-chain worst-case scenario

Acknowledgments

The work originated from a collaboration between Aarhus University, Washington University in St. Louis, and Uber, published as "Dynamic Partial Deadlock Detection and Recovery via Garbage Collection" (Saioc et al., ASPLOS 2025). Google Go team members Michael Knyszek, Michael Pratt, and PJ Malloy guided the transition to a production feature.

Supplementary Examples

Real-world leak patterns from industry codebases, ordered by complexity:

Duplicate Send

An error branch sends on a channel but misses a return, causing a second send that blocks forever. The profiler highlights the leaking send line.

Early Return

The receiver returns early on error, leaving the sender blocked on an unbuffered channel. Fix: add a buffer of size 1.

Timeout with Context

A select waits for either a worker message or context cancellation. If context fires first, the worker send leaks. Fix: buffer the channel.

Range Over Channel Never Closed

Workers loop with for item := range ch but the channel is never closed, so workers block forever after items are exhausted. The profiler shows all workers blocked on range ch. Fix: close(ch) after sending all items. An additional latent leak appears if workers == 0 (sender blocks), which the profiler also catches.

Method Contract Violation

A Worker interface requires Start() followed by Stop(). Calling Start() without Stop() leaves the internal goroutine blocked on a select over ch and done. The profiler reveals the blocked goroutine in Start.

CockroachDB: Missing Unlock Before Break

A loop locks a mutex, checks a condition, and breaks without unlocking. The profiler shows the goroutine blocked on Lock() at the line with the missing Unlock().

etcd: Unexpected Channel Ordering

Concurrent Status(), run(), and Stop() can race such that Status() sends on a channel that is never received because Stop() and run() synchronize and exit first. Fix: wrap the send in a select with a case <-done: to allow graceful exit.

Kubernetes: Channel and Mutex Deadlock

WriteFrame()

holds a mutex and sends on resetChan; monitor() receives on closeChan then tries to lock the same mutex. Because resetChan has no receiver, the send blocks forever, preventing the mutex release. Both goroutines leak. Fix: drain resetChan in a separate goroutine before locking.

Moby: WaitGroup Wait Inside Loop

group.Wait()

is called inside the loop that spawns workers, so the first iteration blocks before spawning the rest. The profiler shows the init goroutine blocked on Wait(). Fix: move Wait() after the loop.

Moby: Channel and Mutex Deadlock (Second)

StateChanged()

holds a container mutex and sends on a stop channel; monitor() has a select with a default case that calls handleProbeResult() which tries to lock the same mutex. If the send blocks, the mutex is never released, and monitor() blocks on the mutex. Both leak. Fix: close the stop channel instead of sending (close is non-blocking), allowing StateChanged() to release the mutex and monitor() to proceed and exit.

Original Source

Signed-in readers can open the original source through BestHub's protected redirect.

Sign in to view source
Republication Notice

This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactadmin@besthub.devand we will review it promptly.

pprofgarbage collectionproduction debuggingchannelsync.MutexGo 1.27concurrency debugginggoroutine leak profiler
Reader feedback

How this landed with the community

Sign in to like

Rate this article

Was this worth your time?

Sign in to rate
Discussion

0 Comments

Thoughtful readers leave field notes, pushback, and hard-won operational detail here.