How We Built a Service‑Decoupled DevOps Platform for Scalable Cloud‑Native Delivery

Business Background

AiFanFan, a typical B2B SaaS provider, operates multiple product lines (expansion, chat, tracking, insight) and faces intense market competition, demanding high R&D efficiency and quality. The organization is divided into several Scrum teams, each responsible for a specific business domain.

Challenges in the Efficiency System

2.1 Service Explosion Increases Infrastructure Costs

More than 200 active modules with an average of eight new modules per month cause a sharp rise in pipeline, monitoring, and other infrastructure maintenance costs.

2.2 Complex Topology Hinders Issue Localization and Regression Assessment

The intricate service mesh leads to difficulties in evaluating upgrade impact, increased missed regressions, hard online issue diagnosis, and high costs for large‑scale integration testing.

2.3 High‑Frequency Releases vs. Rising Deployment Costs

Over 100 modules are released together, requiring manual control, which is risky and inefficient, especially as release frequency grows.

Overall Improvement Approach

Process & Management Layer

Agile Iteration Mechanism: Focus on user‑value flow and transparency to align team goals.

Requirement Decomposition Management: Standardized, visualized, and automated handling of small‑batch demands to accelerate value verification.

Branching Model & Environment Management: Leverage Istio‑based traffic control for lightweight, flexible, low‑risk branching.

Full‑Process Data Measurement: Use objective metrics to assess current state, discover problems, auto‑create tasks, and drive issue closure.

Technical Layer

Infrastructure: Build services that are decoupled from business logic.

Automation: Implement a layered automation framework suitable for microservice architectures.

Release Capability: Provide one‑click, visualized, observable, and controllable release experiences.

Tool Empowerment: Offer rich tooling to address efficiency pain points across development and testing.

Four Technical Directions

4.1 Infrastructure Standardization

Standardize modules (code structure, packaging, container images), pipelines, core services (APM, config center, release platform, resource management), and development models to enable scalable service‑oriented infrastructure.

4.2 Declarative Infrastructure

Provide a one‑click, minute‑level onboarding experience via a scaffolding tool that automatically generates code frameworks, integrates standard components, creates pipelines, provisions clusters, and generates configuration files based on declared module attributes. New services can be fully provisioned and deployed to a test cluster in under ten minutes.

4.3 Intelligent Infrastructure

Introduce strategy‑driven “supervisors” into CI/CD pipelines to automatically decide whether to skip, queue, or retry tasks, thereby improving stability and efficiency. Typical scenarios include automatic red‑light analysis, queue strategies that pre‑check environment health, and configurable policies for task handling.

Layered Automation System

Adopt a reversed‑pyramid automation model where end‑to‑end testing is emphasized due to simple services but complex topology. Automated DIFF testing, contract testing, and front‑end DIFF testing run without human intervention, forming the core of the automation stack.

5.1 Full‑Link Gray‑Scale DIFF Testing

Utilize Istio’s flexible routing and a custom CRD operator to build a gray‑scale release platform that supports multi‑route environments, capacity evaluation, and canary releases. Traffic replay between the base version and a new branch enables automated regression detection.

5.2 Contract Testing to Safeguard Service Calls

Adopt a hybrid contract‑testing approach: the provider generates contracts, while consumption patterns are inferred from logs and call‑chain analysis to automatically create consumer‑side test cases. The workflow includes:

Integrate Swagger to keep API documentation in sync with code.

Automatically generate contract test cases from API specs.

Analyze call‑chain and logs to synthesize consumer contracts and link them to the provider’s APIs.

5.3 Intelligent Issue Localization

When automated test cases fail, an auto‑localization service tags failures, categorizes them (environment, batch unknown, element not found), and triggers appropriate remediation such as retries, escalation to QA, or configuration‑driven handling.

Efficient and Safe Continuous Release

6.1 Release Challenges

Different modules use varied release platforms and processes, making unified deployment difficult.

High‑risk manual control for releasing 100+ inter‑dependent modules.

Lack of visibility into the overall release process.

6.2 Multi‑Platform Deployment Engine

Build a cloud‑native, unified deployment and release engine that integrates seamlessly with CI/CD pipelines, standardizes release procedures, and abstracts underlying platform differences.

6.3 Release Playbook Design

Automate the entire release workflow by collecting data such as module freeze status, dependencies, and configuration, generating a release topology and step list, confirming with humans, and then invoking the release services automatically while recording metrics for post‑release analysis.

6.4 Visualized, Perceptible, Controllable One‑Click Release

Provide real‑time service‑level dependency topology and progress visualization, combined with APM and canary strategies to ensure safe, lossless releases.

Overall Benefits

Story count increased by 85.8%, release cycles stabilized, development‑test cycle shortened by 30%, and bug density dropped from 1.5 to 0.5 per thousand lines of code.

Future Outlook

Integrate IDE plugins to empower developers during coding and testing, further boosting efficiency.

Leverage white‑box capabilities to build a quality‑risk identification system for admission, egress, and gray‑scale scenarios.