Root Cause Analysis and Resolution of Service Availability Fluctuations in a High‑QPS Go Backend

Background : A high‑traffic service handling over 100k QPS experienced occasional availability dips below 0.999, causing timeouts in a critical reservation API during peak periods.

Initial Hypotheses : Potential causes considered included business logic performance issues, abnormal traffic, CPU throttling, middleware (Redis, MySQL), Go scheduler, and GC problems.

Investigation Steps :

Collected pprof data (CPU and memory) to check for CPU hotspots and GC anomalies.

Enabled GODEBUG=gctrace=1 to capture GC traces.

Added fgprof to detect off‑CPU time such as I/O, locks, or paging.

Captured Go trace files to examine scheduler behavior, GC, and pre‑emptive scheduling.

Used strace on the process to inspect system calls.

Analysis Findings :

gctrace : Concurrent mark and sweep phases consumed ~860 ms, but no clear increase compared to stable periods.

strace : No abnormal system calls were observed.

fgprof : No off‑CPU bottlenecks detected.

Go trace : Repeated MARK ASSIST events indicated noticeable GC activity.

Heap analysis : The gcache LFU implementation generated >1 M objects, accounting for a large portion of in‑use objects and memory.

Root Cause : A memory leak in the gcache library’s LFU Get method (see https://github.com/bluele/gcache/issues/71) caused excessive object allocation, leading to GC pressure and request timeouts.

Resolution : Upgraded gcache from v0.0.1 to v0.0.2, eliminating the LFU memory leak and restoring service availability.

Summary & Lessons :

Third‑party library bugs can critically impact system stability; maintain vigilant monitoring of upstream issues.

Comprehensive observability (system, component, and Go runtime metrics) is essential for rapid root‑cause analysis.

Automated profiling (e.g., periodic pprof/trace collection) helps capture transient incidents.