Design and Implementation of a Simple Gray Release System

Definition of Gray Release

Internet products need rapid iterative development and deployment while ensuring quality; a gray release system allows newly launched systems to be quickly isolated if problems arise, essentially functioning as an A/B testing mechanism.

Simple Gray Release System Design

The simple architecture (illustrated in the original diagram) consists of three essential components:

Strategy configuration platform that stores gray release policies.

Execution program that applies gray release functionality.

Service registry where services register with ip/port/name/version.

These three components together form a complete gray release platform.

Gray Release Strategies

Common gray release strategies include:

Traffic splitting based on request headers.

Traffic splitting based on cookies.

Traffic splitting based on request parameters (e.g., user UID modulo for a 1% rollout).

Strategies can be single (e.g., based on UID, token, IP) or composite (multiple services gray‑released together using a tag field).

Specific Execution Control of Gray Release

In the simple architecture, upstream services (the executors of gray policies) can be Nginx or gateway/business/data‑access layers.

Nginx

If the upstream service is Nginx, Lua extensions are needed to implement gray policy configuration and routing, as Nginx does not natively support gray policies. An agent can be deployed locally to receive policies from the configuration platform, update Nginx configuration, and gracefully reload Nginx.

Gateway/Business/Data‑Access Layers

These layers only need to integrate the configuration platform's client SDK to receive gray policies and execute them via the SDK.

Complex Gray Release Scenarios

Scenario 1: Simultaneous gray release of multiple services in a call chain. The solution tags requests with a tag T at the new‑version gateway; downstream services forward requests with tag T to the new data‑access service and others to the old version.

Scenario 2: Gray release involving data changes. Since databases lack a gray concept, data must be duplicated for read/write. Full‑copy data migration is required, with writes duplicated (dual‑write) and a message queue used to ensure consistency after offline full‑copy.

During the gray process, data from the two databases should be compared to verify consistency, ensuring no data loss whether the gray rollout succeeds or is rolled back.