How Gray Release Enables Safe, Rapid Feature Rollouts in Production

Definition of Gray Release

Internet products need fast iterative development and deployment while guaranteeing quality; a gray release system directs a portion of user traffic to a newly launched system for quick validation and immediate rollback if problems arise, essentially functioning as an A/B testing platform.

Simple Gray Release System Design

The basic architecture includes three essential components:

Strategy configuration platform that stores gray release policies.

Execution program that applies the gray release logic.

Service registry that records service IP, port, name, and version.

All three together constitute a complete gray release platform.

Gray Release Strategies

Common strategies include:

Traffic splitting based on request headers.

Traffic splitting based on cookies.

Traffic splitting based on request parameters, e.g., using a user UID modulo operation to allocate 1% of users to the new version.

Gray release policies are divided into single and composite strategies.

Single strategy: split by UID, token, or IP.

Composite strategy: apply multiple services simultaneously, using a tag field to indicate which services participate in the gray release.

Execution Control of Gray Release

In the simple architecture, upstream services execute the gray strategy (e.g., Nginx, gateway, business logic layer) while downstream services provide the actual functionality. Different upstream implementations are discussed below.

Nginx

When Nginx is the upstream, Lua extensions are required to implement gray strategy configuration and routing because Nginx alone cannot execute the policies.

The solution is to deploy a local agent that receives policies from the configuration platform, updates Nginx configuration, and gracefully restarts Nginx.

Gateway / Business Logic / Data Access Layers

These layers only need to integrate the configuration platform's client SDK, receive gray policies, and execute them via the SDK.

Complex Gray Release Scenarios

Two examples illustrate more advanced use cases, both assuming a 1% UID‑based gray split.

Scenario 1: Simultaneous Gray Release of Multiple Services in a Call Chain

The upgrade affects several services; the gateway and data‑access layers are gray‑released while the business logic layer remains unchanged.

Solution: the new gateway tags all requests with tag T; downstream services forward requests with tag T to the new data‑access service and others to the old version.

Scenario 2: Gray Release Involving Data Changes

When new versions introduce additional database fields, the old database cannot be modified directly. The solution is to copy data to a new database and perform dual writes.

During offline full‑copy, some data loss may occur; therefore, the business layer writes data to a message queue (MQ). After synchronization, the new data‑access layer consumes the MQ and writes to the new database, ensuring consistency.

During the gray period, data from both databases should be compared to verify consistency, ensuring no data loss whether the gray release succeeds or is rolled back.