Mastering Gray Release: Real-World Testing Practices from ICBC

Share the testing practices of ICBC Software Development Center project team on gray release, including test shift, static testing, traffic strategy verification, and rollback verification.

1. Shift Testing Forward

Testers participate in defining gray objectives and design, aiming to uncover defects with minimal changes.

(1) Analyze Gray Objectives

The team defines gray deployment strategy and continuous delivery strategy, illustrated in the diagram.

Gray Deployment Strategy : two methods – physical gray and logical switch. Physical gray offers higher safety but higher complexity; a combination is recommended.

Physical gray nodes are deployed via custom routing in a distributed framework; logical switch is used where physical gray is infeasible.

Physical Gray Node Deployment Scope : only service nodes.

Physical Gray Deployment Strategy : single‑version release; if incompatibility exists, use logical switch only.

Physical Gray Version Release Scope : both official and patch versions follow physical gray release.

(2) Analyze Gray Design

Design includes traffic routing, client, server, database, and service continuity.

Traffic Design : design based on channels and user characteristics, using whitelist and progressive rollout to avoid affecting important customers.

Client Design : specify gray version download method per client type.

Server Design : configure gray traffic switches and rules via parameters for real‑time adjustments.

Database Design : evaluate feasibility of physical gray based on schema changes.

Service Continuity Design : adopt fully non‑stop solution ensuring in‑flight business can close loop.

2. Conduct Static Testing

Includes static verification of overall gray strategy and specific feature gray testing.

Overall strategy: ensure physical gray nodes are isolated, use independent container templates, verify node mapping.

Feature gray: evaluate need for gray, feasibility, and specific strategy per feature.

3. Execute Gray Testing

Focuses on traffic testing, gray‑to‑production testing, rollback testing, and non‑business functional testing.

Traffic Testing : validate routing rules and functional correctness after traffic split; automate with scripts scheduled via Jenkins.

For existing features, replay logs on gray and non‑gray nodes and compare results.

Gray‑to‑production testing verifies seamless transition without downtime; rollback testing ensures in‑flight business can be reverted; non‑business tests cover high‑availability and performance.

Continuous quality guarding of gray releases is a long‑term effort requiring ongoing monitoring and improvement.