How a Chinese Telecom Built an Automated MySQL Management Platform

China Mobile’s "Mobile Cloud" serves roughly 150,000 customers with about ten cloud products, and its database instances have doubled in two years while DBA headcount remained flat. To cope, the team transitioned from manual operations to a standardized, tool‑based, and eventually fully automated platform called the EcloudDB Database Management Platform.

1. Standardization

Standardization was the foundation, covering OS‑level settings (kernel, RAID, NTP, SELinux, NUMA, etc.) and MySQL configurations (directory layout, user permissions, my.cnf parameters, account policies, logging, thread pools, backup tools, cron jobs). A unified directory structure enables scripts to perform batch operations across instances.

2. Tooling and Scripting

Early automation leveraged Python and web technologies, then incorporated open‑source tools:

Zabbix for comprehensive monitoring and alerting.

ELK stack for log collection, analysis, and fault tracing.

Ansible for batch deployment and configuration.

Percona Toolkit and XtraBackup for log parsing, online schema changes, and physical backups.

Custom scripts were also created for periodic inspection, backup validation, data consistency checks, and standardization verification.

3. Automated Deployment Process

The deployment workflow uses Ansible roles and playbooks:

Allocate physical or virtual machines based on the deployment plan.

Install Ansible on the control host.

Perform OS initialization (OS version, filesystem, RAID, NTP, SELinux, etc.).

Upload the Ansible role package and the standardized bcrdb package to the appropriate directories.

Prepare an inventory file with cluster node addresses and business‑specific parameters (max_connections, innodb_buffer_pool_size, wait_timeout).

Execute the playbook, e.g., ansible‑playbook -i host playbooks/bcrdb.yml, which pushes the database, runs standardization checks, and registers the instance.

4. Platform Architecture

The platform follows a three‑layer design:

Interaction layer : Web UI and RESTful APIs for user actions.

Service layer : Django‑based backend handling business logic, asynchronous tasks via Celery, and routing to appropriate views.

Resource layer : Data services (MySQL, Redis), OS resources, and auxiliary components (logging, monitoring).

Key technologies include Vue.js for the frontend, Django for the backend, Redis for caching, MySQL for persistent storage, Django‑Celery for long‑running tasks, and Ansible for remote operations.

5. Core Platform Features

Index dashboard showing instance health, performance metrics (TPS, QPS, thread count) and backup status.

User management with role‑based permissions and a workflow for DB account requests.

Backup management supporting on‑demand snapshots, point‑in‑time recovery, and configurable retention policies.

Inspection module offering one‑click batch checks, a library of 40+ health items, and customizable checks.

SQL review using the open‑source YearningSQL engine with added custom rules.

Instance management for creating databases, tables, stored procedures, triggers, and handling permission changes.

6. Future Roadmap

Phase 2 will add log management, slow‑query analysis, lock‑conflict diagnostics, advanced information queries, and enhanced alerting.

7. Lessons Learned

Keep designs simple and avoid over‑engineering; adopt agile, incremental delivery; manage requirement changes carefully; and enforce rigorous testing before production rollout to ensure stability and performance.