How Meituan Built a Scalable MySQL Inspection System to Keep Databases Healthy

Database inspection is essential for ensuring stable and efficient operation by detecting hidden risks early. This article introduces Meituan's MySQL inspection system, describing its overall architecture, design principles, core components, inspection items, and achieved outcomes.

Background

Inspection works such as power or fire checks keep environments stable; similarly, database inspection reduces risk and improves service reliability. Traditional inspection relied on a central control machine, timed scripts, and a front‑end, which introduced single‑point failures, scattered results, inconsistent scripts, and cumbersome UI updates.

Design Principles

Stability : The inspection tool itself must be reliable.

Efficiency : Simplify usage, lower learning cost, and enable rapid deployment of new checks as requirements evolve.

Operability : Store inspection data centrally to drive risk remediation, track trends, and prioritize actions.

System Architecture

The system is divided into three layers:

1. Execution Layer

Inspection Execution Environment : Multiple execution machines run the same scripts, pulling the latest version from a Git repository using Python virtualenv and Git.

Task Scheduling : Uses Meituan's distributed scheduler Crane to avoid single‑point failures; tasks are randomly assigned and re‑assigned on failure.

Inspection Targets : Covers production MySQL instances as well as HA components, middleware, and other surrounding services.

2. Storage Layer

Inspection Database : Stores discovered risks with automatic enrichment (responsible person, detection time), idempotent inserts, and support for semi‑structured results from different inspection types.

Inspection Script Git Repository : Central repository for all inspection scripts, providing common utility functions to lower development effort and ease migration of legacy scripts.

3. Application Layer

Integration with DB Ops Platform : Shows risk details, allows configuration of new inspections, and manages whitelist entries.

Risk Operation Backend : Generates reports on risk trends, stock/increment distribution, and average remediation cycles; includes a reminder system (messages, alerts) to prompt DBAs.

External Data Service : Exposes risk data to other internal platforms such as the "XianZhi" risk‑discovery platform and weekly ops reports.

Inspection Items

Items are divided between DBA‑owned (core components, service stability) and RD‑owned (schema design, usage violations). A total of 64 items are grouped into categories such as Cluster, Machine, Schema/SQL, and HA/Backup/Middleware/Alert. Sample items and their purposes are illustrated with diagrams.

Results

After nearly a year of operation, the system runs 49 new inspection items, has resolved over 8,000 critical risks, and maintains an average remediation time under four days. Risk volume has steadily declined, and integration with the XianZhi platform has driven more than 5,000 RD‑handled risks.

Future Plans

Enhance automation with CI and audit pipelines.

Improve operability by refining risk severity scoring and decision support.

Develop automatic risk remediation capabilities.