How Meizu Built a Continuous Delivery Platform to Boost Ops Efficiency

Meizu's operations team improved their ability to handle change and risk by building a continuous integration platform that delivers high‑efficiency delivery experiences to users and product teams.

<code>Proactively responding to change and risk is a crucial operational capability; Meizu's ops team enhanced this by constructing a continuous integration platform that raises responsiveness, achieves efficiency gains, and provides the most effective delivery experience for users and product teams.</code>

Internet Development and Operations Challenges

Meizu History

2003‑2008: Internet 1.0 era – Meizu was founded as an MP3 manufacturer with only a website and BBS.

2009‑2010: Internet 2.0 era – Transitioned to smartphones, adding e‑commerce, cloud services, and establishing real operations and backend development.

2012‑2013: Internet 2.5 era – Android firmware updates, fragmented databases, MFS storage, and GSB scheduling.

2014 onward: Internet 3.0+ era – Game center, app store, multimedia, O2O; cloud platform and big data support an open shared ecosystem.

Challenges from Era Changes

The shift from 1.0 to 3.0 introduced hardware, architectural, and availability issues, low monitoring coverage, unreliable alerts, fragmented processes, lack of capacity planning, and a culture of firefighting, prompting a re‑evaluation of ops value.

Quality : Measured by availability (network, system, service) and experience metrics such as speed and SMS delivery rate. Efficiency : Assessed via server provisioning, change management, releases, scaling, fault detection and resolution. Cost : Evaluated through resource utilization (CPU, network, storage) and labor costs. Security : Enforced via policies, data classification, encryption, logging, isolation, and backup.

Four dedicated teams address these dimensions: quality optimization, efficiency development, cost management, and security operations.

Meizu Operations Platform

Resource Management System

Built a business‑tree, KVM and Docker based cloud platform, providing virtual compute, network, and storage resources managed through CMDB.

Configuration Management System

Developed management systems for iOS, CDN, DNS and exposed APIs.

Automation System

Implemented ticketing, logging, release, application management, custom ops channel, and automated inspection to improve delivery and change efficiency.

Monitoring and Capacity Management

Established baseline monitoring, business‑level monitoring, and a capacity model to track server, memory, bandwidth, and data‑center utilization.

Security System

Deployed bastion hosts for controlled access, WAF, and a vulnerability management system that automates detection, remediation, and scoring.

Meizu Continuous Release Platform

Release Platform Evolution

Moved from manual weekly releases to daily and self‑service releases; early architecture issued commands to service machines, later added metrics, audit trails, and higher success rates (over 98%).

Current features include self‑service, group releases, one‑click restarts, static file deployment, and support for Jetty, task, static workloads.

The delivery flow spans development, testing, and production environments, with automated deployment, logging, alerting, and rapid scaling.

Release Process Issues

Quality gaps (unit test coverage, bugs), efficiency gaps (disconnected automation), cost gaps (high communication overhead), and security gaps (code security testing) were identified.

Continuous Delivery Platform Evolution

Standardization

Defined standards for servers, logs, monitoring, technology stack, services, and automated testing.

Automation

Stage 1: CI pipeline – requirement intake, code review, build, static analysis, automated testing (functional, security, performance), and production release with gray‑release and verification.

Stage 2: Release workflow – environment checks, artifact retrieval, monitoring pause, web traffic drain, service stop, file update, service start, health check, web bring‑up, monitoring resume.

Stage 3: Product development model – supports rapid iteration, parallel development‑test‑release cycles, and post‑release retrospectives.

Value‑Driven Data Output

Collects metrics on code quality, readability, test coverage, performance, security, personnel success rates, and release success/efficiency to drive continuous improvement.

Future – Intelligent Operations

Advances toward intelligent ops using supervised and unsupervised learning for prediction and analysis, such as hardware failure forecasting, automated fault detection, localization, and proactive remediation.