How to Build a Simple Gray Release System for Safe Feature Rollouts

Internet products require rapid iteration while maintaining quality; a gray release system lets newly launched features be tested on a controlled user segment and quickly rolled back if problems arise, essentially acting as an A/B testing platform.

Simple Gray Release System Design

The basic architecture consists of three essential components: a strategy configuration platform that stores gray release policies, an execution program that applies the policies, and a service registry that records service IP/Port/name/version. Together they form a complete gray release platform.

Gray Release Strategies

Common strategies include traffic splitting based on request headers, cookies, or request parameters. For example, using a user ID modulo operation to direct 1% of users to the new version while the rest stay on the old version. Strategies can be single (e.g., based on UID, token, IP) or combined (multiple services gray‑released together using a tag field).

Execution Control of Gray Release

Upstream services (e.g., Nginx, gateway, business logic, data access layers) execute the gray policies. Downstream services receive the routed traffic.

Nginx

When Nginx is the upstream, Lua extensions are used to implement gray policy configuration and routing, with a locally deployed agent receiving policies from the configuration platform and updating Nginx gracefully.

Gateway/Business Logic/Data Access Layers

Integrate the configuration platform's client SDK to receive policies and execute them directly in the service code.

Complex Gray Release Scenarios

Scenario 1: Simultaneous gray release of multiple services in a call chain. The new gateway tags requests with a tag T; downstream services forward requests with tag T to the new data access layer, while others go to the old version.

Scenario 2: Gray release involving data changes. Since the new version may have different database schemas, data must be copied to a separate database for reads and writes (dual‑write). An offline full data copy may lose data, so the business layer writes changes to a message queue; after synchronization, the new data access layer consumes the queue to ensure consistency. Comparing data between the two databases verifies integrity before fully switching.