How Vipshop Scaled DevOps: From Chaos to Standardized Operations

Former Problems

Vipshop, a mid‑size internet company, faced many operational blind spots because its business spans e‑commerce, logistics, and finance, leading to a chaotic mix of technologies and non‑standardized processes. Different teams worked on separate business lines, causing difficulty sharing manpower and a lack of unified platform support.

Although numerous platforms for release and change existed, they felt fragmented and could not provide strong, cohesive support for business teams.

Key questions emerged:

How can technical staff balance quality, cost, and efficiency ?

Why do operators remain exhausted despite many tool platforms?

How to keep the original DevOps mission while sprinting through platform construction?

Standardization Roadmap

Component Concept

The component model establishes a foundation for standardization:

Technical growth is driven by a component expert group that defines direction and best practices, fostering skill development.

Service‑oriented architecture enables operators to deliver services while developers use standard APIs without worrying about underlying details.

Eliminating business silos creates new aspirations for product teams.

Standardization Blueprint

The blueprint splits a large DevOps standardization project into dozens of sub‑projects, aligning technical stacks on the left with concrete outputs on the right; red items indicate business‑specific components, while lower‑level items relate to operations.

Configuration Library Management

Developers often need to change production configurations, which creates friction between developers and operators. The solution is layered governance: developers handle business‑logic changes, while the expert group manages component‑level configurations. A “Janitors” platform provides a standardized view of configuration files, granting developers controlled access to modify parameters safely.

Operators focus on the lower layer, using Puppet for configuration management.

Monitoring Standardization

When the number of machines exceeds 10,000, Zabbix becomes overloaded; Vipshop initially ran multiple Zabbix instances.

Ideal monitoring should be unified, fast, precise, have a single entry point, be automated, and avoid manual intervention. Existing Zabbix solutions did not meet these criteria.

Vipshop built VIPFalcon (based on OpenFalcon) with ~25,000 nodes and over 5 million metrics, using plugin‑based data collection and Hive for analytics, delivering a single pane of glass for all baseline monitoring.

Change Standardization

Two ideas drive change management:

A risk matrix (implemented as an SDK) evaluates change risk from both object importance and technical risk dimensions, providing a precise score.

A standardized change‑template library, designed by component experts, ensures changes follow approved patterns rather than ad‑hoc solutions.

Standardized change apps are now available on a change platform for one‑click execution.

Ecosystem Integration and Full Empowerment

The goal is to let systems drive other systems via API calls, minimizing human intervention.

CMDB Replaces Rigid Processes

Traditional workflow‑centric processes are inflexible; integrating CMDB with operational workflows enables smarter decision‑making based on real‑time asset information.

When a monitoring alert (e.g., disk usage > 90 %) triggers, an automated app cleans the disk and notifies the team, achieving a near‑real‑time closed loop.

Change Integration Flow

SDKs expose change‑control capabilities to automation tools, while a central platform aggregates risk scores and enforces standardized templates, allowing low‑risk changes to be applied instantly by developers.

This reduces approval cycles from half a day to a single click, delivering higher quality, faster delivery, and controlled risk.

Monitoring‑Driven Automation

All server lifecycle actions (init, deploy, run, pause, retire) are triggered by CMDB events consumed by the monitoring system, eliminating manual steps.

After standardization and ecosystem integration, Vipshop gains comprehensive data for AIOps, improves efficiency while maintaining risk controls, and achieves measurable gains in quality, speed, and cost.