Inside ICBC’s Gray Release System: Reducing Risk with Canary, Rolling & Blue‑Green Deployments

1. Industry Status

To limit the impact of version defects, the industry commonly adopts progressive release processes, expanding the gray‑release user base step by step (whitelist → internal group → specific production users → all users). This staged rollout enables comprehensive quality verification while keeping issues confined to a small audience for rapid response and rollback.

Different deployment modes allow flexible control of the release scope. The main approaches are:

1. Canary Deployment

Update a small fraction (e.g., 2%) of servers first, test the application, and if validation passes, continue updating the rest; otherwise roll back. This limits impact and is suitable when confidence in new features is low or high availability is required.

2. Rolling Deployment

An enhanced version of canary deployment that gradually expands the gray‑release range after successful verification, updating servers in batches (e.g., 10%, 30%, 100%). It requires automated deployment tools and routing mechanisms to ensure a smooth user experience, with longer deployment and rollback times.

3. Blue‑Green Deployment

Maintain two identical production environments (blue and green). Only one serves traffic while the other stays idle. New versions are deployed and validated in the idle environment, then traffic is switched via routing, making the previously active environment idle for the next cycle.

2. ICBC’s Gray Release System Construction

Based on recent exploration and practice, ICBC built a gray‑release system tailored to its architecture, covering standard specifications, management processes, and automation tools.

1. Gray‑Release Specification

ICBC defines technical specifications and best‑practice guidelines for gray releases across design, operation, and lifecycle stages. The specifications include basic principles, target setting, user‑strategy design, traffic routing, validation, rollback, and upstream/downstream coordination, forming a closed‑loop lifecycle management.

Best‑practice guidance addresses architecture design for front‑end apps, back‑end servers, and databases, as well as full‑transaction‑link gray release and automation design, reducing implementation cost and improving standardization.

2. Gray‑Release Management制度

The implementation spans the entire project lifecycle, from requirement definition to production rollout. ICBC established an independent gray‑release management制度, clarifying roles and responsibilities and setting requirements from a R&D management perspective.

3. Supporting Systems

Two main systems support gray releases:

(1) Online Control System

During requirement development, this system tags requirements for gray release, propagates the tags through development and production, and carries them to the gray‑release stage, solving cross‑region and cross‑department information silos and improving communication.

(2) Application Gray‑Release System

Built on a developer portal, build cloud, integrated pipeline, and intelligent verification platform, it automates the entire gray‑release flow from development testing to full production, enhancing delivery capability.

The developer portal integrates project management, version control, automated quality gates, and branch management, triggering build pipelines, unit tests, Sonar scans, coverage statistics, and quality checks before merging code into release branches.

The Build Cloud modularizes build scripts into reusable images, allowing applications to assemble their own build process and providing standardized build services for various technology stacks.

The integrated gray‑release pipeline combines version rollout, verification, pilot scope adjustment, and full release, extending the continuous delivery pipeline from deployment day to gray‑to‑full conversion day, achieving end‑to‑end automation.

The Intelligent Verification Platform automates validation across logs, databases, services, APIs, files, batch jobs, and configuration centers, integrating with a distributed configuration center for full‑scale automated verification.

3. Future Outlook

ICBC’s gray‑release system now plays a crucial role in risk control and financial stability. As the smart‑banking ecosystem accelerates, higher demands will be placed on gray‑release capabilities. Future work includes:

1. Standardized Implementation

Establish a “release unit + service environment” layered deployment mechanism to support simple structured deployment descriptions and multi‑batch releases, gradually standardizing existing gray‑release systems.

2. Enhancing Platform Capabilities

Improve evaluation, monitoring, and dashboard features to reach enterprise‑level, high‑efficiency automated gray‑release management.

3. Boosting Business Innovation

Leverage the mature gray‑release framework for user‑experience and innovation experiments such as A/B testing, collaborating with business units to enhance customer satisfaction.