Meizu's Automation Journey and Continuous Delivery Platform Evolution

Meizu's internet development is divided into three eras: 2003‑2008 (Internet 1.0) focused on MP3 players with a single IDC; 2009‑2011 (Internet 2.0) introduced smartphones, cloud services, and basic master‑slave database replication; 2012‑2013 (Internet 2.5) added Android, e‑commerce, micro‑services, Redis clusters, and distributed storage, while 2014 onward (Internet 3.0+) made internet services a core business.

Rapid growth across these eras created operational challenges in quality (poor monitoring, low coverage), efficiency (manual processes, low automation), cost (lack of capacity assessment), and safety, leading to frequent firefighting and the need for a value‑oriented operations strategy.

The operations platform now consists of several subsystems: a cloud‑based resource management system built on KVM/Docker with CMDB; configuration management (LVS, CDN, DNS) with fine‑grained permissions; automation tools for work orders, logging, releases, and self‑service pipelines; comprehensive monitoring (basic, custom, business, capacity); a bastion host for access control; and a self‑developed WAF for vulnerability management.

The release platform evolved from manual weekly releases to automated daily and self‑service deployments, introducing standards, business‑level approvals, and a variety of release strategies (group, self‑service, one‑click, static files) that now achieve >98% success rates with ~90% of services released without operations involvement.

The delivery workflow spans development, testing, and production environments, integrating Git, Jenkins, static analysis, automated testing, and operations‑provided infrastructure, while emphasizing strict process adherence to maintain balance and prevent delivery breakdowns.

Key delivery problems identified include code quality (unit test coverage, bug count), efficiency (disconnected automation across teams), and security (lack of thorough testing), prompting the formation of dedicated teams for quality, efficiency, cost, and security.

The continuous delivery platform follows a standardized, automated, and intelligent progression, with standardization of monitoring, tech stacks, and hardware; automation of testing and CI pipelines; and intelligent operations that leverage collected data for predictive analysis, fault detection, and capacity planning.

Two implementation options were considered: a fully open‑source stack (Docker, Elasticsearch) versus extending existing in‑house systems; the latter was chosen despite higher integration resistance.

A unified entry point (e.g., Jenkins API) aggregates requirements, bug tracking, and personnel data, enabling multi‑user coordination across development, testing, and operations.

The continuous integration pipeline moves from requirement gathering, development, static scanning, testing (including security and performance), to production release with gray‑release validation, health checks, and monitoring re‑enablement.

The release process includes environment verification, artifact retrieval, monitoring suspension, web traffic shutdown, service stop, file update, service restart, health verification, web re‑enablement, and monitoring restoration, supporting both serial and parallel deployments.

With the platform in place, Meizu can support rapid iteration models, capture extensive quality and performance metrics (code quality, test coverage, performance, security, release success, efficiency), and use these data‑driven insights to improve engineering practices, assess subsystem health, and predict infrastructure failures.