After a high‑traffic Go service’s availability slipped below the 99.9% target, the team traced the issue through pprof, gctrace, strace, fgprof, and heap analysis, ultimately identifying a memory‑leak bug in the third‑party gcache LFU implementation and resolving it with an upgrade.
This article details the systematic investigation of intermittent availability drops in a high‑throughput Go service, covering hypothesis formulation, extensive profiling with pprof, gctrace, strace, fgprof, go trace, heap analysis, the discovery of a gcache LFU bug, and the final remediation steps.
A high‑traffic Go service suffered intermittent availability drops below 99.9% due to timeouts, and after extensive profiling, tracing, and heap analysis the root cause was identified as a memory‑leak bug in the third‑party gcache LFU implementation, which was fixed by upgrading the library.
An investigation of a 100k‑QPS Go monolith revealed that intermittent availability drops were caused by a memory‑leak in the third‑party gcache LFU implementation, which inflated GC work and produced long mark phases; upgrading gcache eliminated the leak and restored 0.999+ availability, highlighting the need for thorough observability and dependency monitoring.
