How Bilibili Scaled Server Fault Management with Automated Detection and Repair

This article details Bilibili's evolving server fault management architecture, covering fault classification, the shortcomings of manual processes, and the design of an automated detection and repair system that combines in‑band and out‑of‑band data collection, rule‑based alerts, and end‑to‑end repair automation.

Operationsin‑band collectionmonitoring

0 likes · 18 min read

How Bilibili Scaled Server Fault Management with Automated Detection and Repair

High Availability Architecture

Jul 22, 2025 · Operations

How We Automated Server Fault Detection and Repair at Scale

This article explains the challenges of managing rapidly growing server fleets, outlines a systematic classification of hardware and software faults, and details an end‑to‑end automated solution that combines in‑band and out‑of‑band data collection, rule‑based detection, and fully automated repair workflows to improve fault coverage, accuracy, and recovery speed.

Operationshardware detectionmonitoring

0 likes · 16 min read

How We Automated Server Fault Detection and Repair at Scale

Bilibili Tech

May 27, 2025 · Operations

Automated Server Fault Detection and Repair: Architecture, Methods, and Future Outlook

This article presents a comprehensive overview of server fault management at scale, detailing the classification of failures, shortcomings of traditional manual processes, and the design of an automated detection and repair system that combines in‑band and out‑of‑band data collection, rule‑based alerting, and end‑to‑end repair workflows, while also outlining future directions for intelligent monitoring and reliability.

AutomationInfrastructureOperations

0 likes · 17 min read

Automated Server Fault Detection and Repair: Architecture, Methods, and Future Outlook