How Qunar Scaled DevOps: From Process to Cloud‑Native Automation

Overview

Qunar has built a large‑scale DevOps ecosystem that follows cloud‑native principles and combines open‑source components (Kubernetes, SonarQube, Jacoco, etc.) with internally developed services. The core identifier APPCODE links every application to its project, enabling end‑to‑end traceability across development, testing, release and operations.

Project Workflow

Business goal definition → product planning → detailed design.

Feature breakdown into requirements → agile development.

Integration testing and verification.

Production deployment and operation support.

DevOps tooling is embedded in each stage to accelerate delivery while preserving quality.

Goals and Methodology

Inspired by Baidu’s engineering capability model, the two primary objectives are:

Increase delivery speed through systematic processes and reusable tooling.

Guarantee delivery quality via automated quality gates and data‑driven monitoring.

The methodology consists of a documented process, a shared platform, and scenario‑driven practices that are applied consistently across all projects.

Ecosystem Architecture

The ecosystem is divided into four domains:

Development : application registration, source code management, static analysis (SonarQube), code review.

Testing : defect tracking, test case management, performance testing, code coverage (Jacoco).

Release : release orchestration, artifact storage, rollback mechanisms, load‑balancing.

Operations : monitoring, logging, event handling, alerting.

Underlying infrastructure is a mix of KVM virtual machines and Kubernetes clusters. Open‑source components are highlighted in blue, internally extended components in yellow, and fully proprietary services in red.

Measured Effects

Typical monthly statistics after ecosystem rollout:

>3,000 releases

>150,000 deployments

2,000+ applications

1,000+ developers, testers and operators

Most releases are self‑tested by developers, demonstrating that the platform reduces hand‑offs and accelerates feedback.

Core Practices

1. Standardized Branch Naming for Project Traceability

Each Git branch must contain a PMO identifier (e.g., feature/PMO1234‑new‑api). When the branch is created or pushed, an automated hook links the commit to the corresponding project in the management system, eliminating manual data entry and providing real‑time status for managers.

2. Multi‑Channel Quality Assurance & Release Gate

Quality is enforced through a combination of:

Code review (Phabricator‑based diff creation and review).

Static analysis with SonarQube, configured for Java‑specific rules (illegal package names, duplicate classes, dependency version conflicts).

Automated interface regression (tool "灭霸") and unit‑level API tests ("Qunit").

Coverage measurement via Jacoco, supporting both incremental and full reports.

Environment management platform that provisions isolated test environments on demand.

Quality gate service Cable that consumes all check results, aggregates them, and blocks releases that do not meet the defined thresholds.

3. Application Portrait (AppCode) for Release & Operations

An application portrait stores:

Unique AppCode (the primary identifier).

Environment attributes (hardware, middleware, network config).

Release parameters (build flags, packaging options, deployment strategy).

Dependency information (service calls, database connections, domain owners).

The portrait is consumed by a configuration service, triggers Jenkins pipelines, and enables automatic correlation of alerts, logs and traces to the originating application.

4. Continuous Delivery Pipelines

Two pipeline models are defined:

Single‑application pipeline : stages for development → unit test → integration test → production deployment. Data such as DB schema changes are propagated automatically.

Multi‑application pipeline : coordinates cross‑application releases, ensuring dependent services are deployed in the correct order and that shared resources are locked during the rollout.

Both pipelines support custom stages for business‑specific steps, allowing teams to extend the standard flow without breaking the overall process.

Future Roadmap

Developer platform : auto‑generation of API client code and service‑governance configuration (e.g., qconfig integration).

Chaos‑engineering framework : systematic fault‑injection and rehearsal of failure scenarios.

Service observability platform : unified health dashboards, automated incident localization, and integration with tracing, logging and metric systems.

Selected Q&A (Technical Highlights)

CI/CD ordering : pipelines expose explicit gate checkpoints; downstream pipelines are triggered only after upstream success, and a unique build identifier links the two flows.

Metric system construction : align indicators with business goals using the MARI principle (Metric‑Analysis‑Result‑Improvement). Separate process metrics (lead time, change failure rate) from result metrics (availability, error budget).

Layered requirement management : adopt OKR cascading – company‑level objectives → team OKRs → concrete feature tickets – to ensure traceability from business goal to code change.

DevOps vs. SRE : DevOps covers the full lifecycle (project management, development, testing, release, operations) while SRE focuses on reliability engineering within the operations domain.