High‑Quality Continuous Delivery Practices in Mobile Taobao: Architecture, Process, and Quality Assurance

Preface

With the rapid spread of mobile Internet, the Mobile Taobao app has grown into a massive product containing over 40 bundles (business modules) and supported by hundreds of developers. The rapid growth in business and staff posed significant challenges to architecture, team collaboration, and product delivery. This article shares the team’s two‑year journey toward high‑quality continuous delivery, presenting experiences and lessons learned.

Phase 1: Single Project, Single Build Artifact, Basic Process, Basic Quality Assurance

Two years ago Mobile Taobao was a young product with few features, a small team, limited code, and simple testing methods. All code lived in a single project; testing and release revolved around the compiled package of that project.

Project Architecture

At this stage the app was essentially a monolithic project where all developers shared the same source tree.

Problems

Code was mixed together, making management difficult; a problem in one module affected the entire team.

The architecture could not support rapid business growth.

Development Process

The simple architecture led to a straightforward process: developers committed locally, the build server packaged the code, testers tested the package, and releases were made based on the latest commit.

Problems

Testing and integration phases were merged, resulting in low‑quality test builds and frequent bug‑fix submissions.

Any compilation failure blocked the whole team.

Rollbacks were limited to code‑level.

Pre‑release regression required waiting for bug fixes and re‑packaging, causing heavy workload.

Critical blockers (e.g., login issues) forced all developers to wait.

Only a single release step increased the risk of defects reaching production.

Quality Assurance Measures

Testing relied mainly on manual testing, supplemented by monkey testing and a few automated scripts.

Problems

Manual testing caused repetitive work for testers.

Testing experience was not accumulated.

Non‑functional issues lacked testing methods.

Uncontrolled integration package cadence forced testers to repeatedly switch packages.

Multiple environment packages (test, pre‑release, production) required repeated installations.

Phase 2: Multi‑Project, Single Build Artifact, Platform Establishment, Basic CI, Release Process Setup

As mobile Internet exploded, Mobile Taobao’s business and team grew dramatically, exceeding 20 bundles and a hundred developers. The original monolithic architecture could no longer meet the needs, prompting the creation of packaging, release, testing, and acceptance platforms, as well as a gray‑release process.

Project Architecture

The architecture shifted to a modular approach where each bundle was built, packaged, and deployed to a repository; a builder project managed dependencies and final packaging.

Problems

All developers still shared a single dependency configuration, so changes in one module could affect others.

Version control in the repository was chaotic; development could deploy arbitrarily, making integration packages uncontrollable.

Full‑package compilation became slow as code volume increased, leading to long wait times and frequent failures.

Bundles still compiled into a single artifact, causing cross‑bundle impact and difficult failure diagnosis.

Typical scenarios included long compilation times for minor changes and prolonged waiting for stable test packages.

Development Process

With the new architecture, integration responsibilities shifted from individual developers to module teams, and integration moved from code commits to repository deployments. Gray‑release practices helped expose issues early, improving final release quality.

Processes such as code review, packaging, release, testing, and sentiment monitoring were automated, improving efficiency.

Quality Assurance Measures

The testing team introduced memory, performance, and traffic‑power tests, built semi‑automated tools, and formed an outsourcing team to increase coverage.

Problems

Testers spent excessive time updating and waiting for packages.

Automated and specialized tests could only be applied after the gray‑release stage, compressing testing time and increasing fix costs.

Non‑functional testing coverage remained low.

Multiple environment packages forced repeated installations.

Current Phase: Multi‑Project, Multi‑Build Artifacts, Mature Process, Comprehensive Quality Assurance

Building on previous phases, the team further refined architecture, processes, and quality measures, establishing the current system.

Project Architecture

The architecture now adopts a plug‑in system with per‑bundle compilation, drastically reducing build time and enabling early detection of compilation issues. Dependency configurations are isolated per bundle, allowing independent development, testing, integration, and release without interference.

Development Process

With solid architectural support, the workflow follows clear boundaries: independent bundle development, testing, integration, gray‑release, and final release. Standardized procedures are enforced by platforms, and quality assurance has significantly improved.

Each bundle has its own independent plan.

The main project defines release schedules and integration windows.

During the integration window, bundles submit independently.

Integration submissions go through checklists, code reviews, bug statistics, and pre‑integration builds.

Daily integration builds provide fresh packages for testers.

Only bundles meeting quality and schedule criteria are integrated.

A candidate package is always available for gray‑release.

Independent bundles can join any release plan based on their needs.

Release cycles are two weeks, with the capability to release within an hour if needed.

Quality Assurance Measures

To maintain high product quality under rapid delivery, the team focuses on four areas:

1. Process

Created testing, integration, and release workflows with automated checks.

Established continuous integration to detect issues early and improve package quality.

Implemented online and offline monitoring and analysis.

2. Package Stability

Bundles control test package frequency; daily builds ensure stable integration packages.

Support for environment switching within a single build eliminates the need for multiple installations.

3. Automated Testing & Tools

Integrated multiple static analysis engines with custom rules (adaptation, crash, framework, security).

Inserted test SDKs into test builds without code intrusion; SDKs are removed for production releases.

Continuous automated testing throughout development improves stability and uncovers performance, power, and other non‑functional issues.

Mock tools and verification platforms boost tester efficiency.

4. Online & Offline Monitoring

Aggregated offline quality data, online business issues, and sentiment feedback for unified analysis and alerting.

Data‑driven insights drive process optimization, test case addition, rule creation, automation scenarios, and new testing tools, forming a closed‑loop quality improvement cycle.

The team’s experience demonstrates that, despite progress in architecture, process, and quality assurance, challenges such as stability, power consumption, traffic, performance, adaptation, user experience, operations, and fault alerting remain in the mobile Internet domain.

Author Biography

Yang Qiang (alias: Yuan Zhan), Alibaba Technical Expert, joined Alibaba Wireless Business Unit in 2012. He leads the Mobile Taobao specialized testing group, focusing on wireless development support platforms, operation platforms, testing tools, and SDK testing. Previously worked at Trend Micro and Huawei in testing roles.