How ICBC Tames MySQL: Real‑World Governance, Risk Mitigation, and SRE Practices

Risk Introduction

MySQL’s free nature brings hidden risks such as pervasive slow‑SQL, which can cause CPU spikes, thread pool exhaustion, and full‑system outages. Real incidents include a single slow query that drove CPU to 100% and forced an emergency master‑slave switchover.

Risk Prevention

ICBC mitigates these risks by enforcing strict design guidelines (e.g., mandatory primary keys, avoiding read‑write separation on stale replicas), establishing metadata standards, and creating tiered training courses for developers, DBAs, and senior engineers.

Methodology

The governance methodology follows four steps:

Standardization : Define concrete rules such as “every table must have a primary key” and quantify acceptable ratios (e.g., rows_examined:rows_sent < 100:1, transaction undo < 100 k).

Quantification : Use precise thresholds instead of vague terms like “a little”; enforce metrics like rows_examined/rows_sent and transaction size limits.

Pitfall Avoidance : Document known MySQL bugs (e.g., TRUNCATE hangs, misuse of REPLACE INTO, and UPDATE … SET … AND … syntax errors) and prohibit risky patterns.

Clarity : Provide clear rationale for each rule so developers understand both the “what” and the “why”.

These rules are integrated into SonarQube via custom plugins that scan mapper.xml files and synchronize local checks with cloud‑based quality gates.

Ensuring Practical Implementation

Automation replaces manual gatekeeping: a quality‑gate system automatically blocks non‑compliant changes, reducing human error and ensuring consistent enforcement across teams.

Production Control

Automated monitoring detects slow‑SQL and large transactions. When thresholds are breached, the system can automatically kill offending sessions, but safeguards prevent blanket kills for batch workloads. Metrics from performance_schema.statements_summary_by_digest and Oracle‑style AWR snapshots are used to quantify query impact.

SRE Management

ICBC adopts a Google‑inspired SRE model, dividing responsibilities into production emergency response, development‑stage governance, and release‑stage validation. Key activities include:

Coordinating emergency incident handling, documenting root causes, and maintaining a checklist for future releases.

Upgrading development processes with automated code quality checks, security scans (SQL injection, unsafe UPDATE syntax), and continuous integration pipelines.

Ensuring release readiness through checklist verification and automated gate enforcement.

Future Vision

The team aspires to a “1‑5‑10” incident response model: detect issues within one minute, pinpoint root cause within five minutes, and achieve self‑healing within ten minutes. Plans include higher‑frequency metric collection (CPU, memory, thread states), advanced anomaly detection using logistic regression and isolation forests, and expanding automated failover capabilities.