How We Built a Scalable DevOps Toolchain for Faster Product Delivery

Background

The organization grew to support over a thousand services and a development staff of several hundred. Rapid requirement changes, legacy process decay, and a shift to container‑based micro‑services created a need for a modern, container‑friendly DevOps toolchain.

Core DevOps Stages

Plan : project‑management activities.

Build (Code) : development activities.

Test : quality‑assurance activities.

Release (Deploy) : combined QA and operations activities.

Monitor : continuous monitoring by development, QA, and operations.

CMDB Design

The Configuration Management Database (CMDB) models the relationships among services , resources , and personnel . The service entity is the central configuration item, linking all other items and providing a unified data model for downstream tools, thereby reducing duplicated maintenance.

Monitoring System Architecture

The monitoring platform is divided into three layers:

Resource Monitoring : Uses open-falcon for host‑level metrics; Grafana dashboards provide common visualizations.

Application Monitoring : Captures availability and performance of front‑end, back‑end, and service calls via a custom Caesar system built on Pinpoint.

Business Monitoring (GoldenEye) : Real‑time, second‑level business‑logic monitoring based on a data‑lake approach. It supports rapid data‑source ingestion, model construction, customizable dashboards, and alert rules.

CI/CD Challenges

Inconsistent branch‑management strategies across teams.

Multiple CI/CD tools (GitLab‑CI, Jenkins) leading to fragmented pipelines.

Low automated‑test coverage, causing manual verification steps.

CI/CD Solution

The approach centers on treating the build artifact as the single source of truth and moving artifact generation to the Continuous Integration stage.

Standardize on gitlab-ci for the entire CI pipeline. Define branch policies (e.g., trunk‑based development with feature‑branch naming conventions) and encode them in .gitlab-ci.yml files (Pipeline‑as‑Code).

Generate immutable artifacts (container images or binary packages) during CI. All downstream QA and production deployments use the same artifact, eliminating “code‑vs‑artifact” drift.

Integrate automated test suites (unit, integration, contract) into the CI pipeline; test results gate promotion to the QA stage.

Adopt the internally built Opera release platform for artifact storage, release planning, and multi‑environment deployment orchestration.

R&D Efficiency Platform (Overmind)

The organization adopted the Overmind platform to provide a unified end‑to‑end delivery pipeline with three primary capabilities:

One‑stop delivery : Teams focus on value flow rather than managing disparate task queues.

Visualization : Real‑time pipeline graphs expose bottlenecks and enforce end‑to‑end quality gates.

Full‑process metrics : Aggregated performance data enable correlation analysis and data‑driven improvement (“If you can’t measure it, you can’t improve it”).

Future Roadmap

Subsequent development cycles will alternate between deepening domain‑specific capabilities and building cross‑domain collaboration platforms. Planned focus areas include:

Risk control during change processes – consolidating change events, defining unified risk metrics, and supporting CD decision‑making.

Environment governance – resolving DevOps environment conflicts and providing consistent sandbox management.

Standardized plugin interfaces – exposing capability APIs for easier extension and integration.