Postmortem of Bilibili SLB Outage on July 13, 2021

On July 13, 2021 at 22:52, Bilibili's SRE team received alarms indicating that the access layer (four‑layer LB and seven‑layer SLB) was unavailable, causing users to be unable to access the website and app. Initial suspicion fell on network or load‑balancer infrastructure, prompting an emergency voice conference.

Investigation revealed that the seven‑layer SLB, built on OpenResty, hit 100% CPU usage due to a Lua bug. The bug originated from a special deployment mode that temporarily set a container instance weight to the string "0". The lua‑resty‑balancer module's _gcd function received this string, performed a modulo operation, produced nan , and entered an infinite loop, exhausting CPU.

Mitigation steps included attempting to reload the SLB, cold‑restarting it, and finally rebuilding a brand‑new SLB cluster. The new cluster was initialized, four‑layer LB and public IPs were configured, and traffic was gradually shifted back, restoring core services (live streaming, recommendation, comments, etc.) by 01:50.

Root‑cause analysis confirmed the Lua type‑conversion issue and the lack of weight validation. Temporary fixes such as disabling JIT compilation helped but did not fully resolve the problem; the underlying weight‑zero condition had to disappear for the service to recover.

Following the incident, several improvements were planned: strengthening multi‑active (multi‑active) architecture, isolating SLB clusters per business unit, version‑controlled Lua code deployment, automated SLB provisioning within five minutes, enhanced monitoring and event‑query capabilities, and formalized incident‑response procedures.

The postmortem concludes that while the outage highlighted serious gaps in load‑balancer reliability and operational processes, the lessons learned led to concrete technical and organizational actions to prevent similar failures in the future.