Design and Implementation of a Gray Release System for Microservices

Definition of Gray Release

Internet products need rapid iteration while ensuring quality; a gray release system directs a portion of user traffic to a newly deployed service to validate new features and quickly roll back if problems arise, essentially functioning as an A/B testing platform.

Simple Gray Release System Design

The basic architecture includes three essential components:

Strategy configuration platform that stores gray release policies.

Execution program that applies the gray logic.

Service registry where each service registers with ip/Port/name/version.

These three components constitute a complete gray release platform.

Gray Release Strategies

Common strategies include traffic splitting based on request headers, cookies, or request parameters. For example, using the user uid modulo 100 to allocate 1% of users to the new version.

Single vs. Composite Strategies

Single strategy: Apply modulo on a single attribute such as uid, token or ip.

Composite strategy: Use a tag field to coordinate multiple services, e.g., applying the same tag T to requests that should be routed to new versions of several services simultaneously.

Execution Control of Gray Release

The system consists of upstream (strategy executor) and downstream services. Upstream can be Nginx, a gateway layer, or business logic layer.

Nginx

When Nginx is the upstream, Lua extensions are used to implement strategy configuration and traffic forwarding. Since Nginx cannot directly receive policies from the configuration platform, a locally deployed Agent updates Nginx configuration and performs graceful restarts.

Gateway / Business Logic / Data Access Layers

These layers only need to integrate the configuration platform’s client SDK to receive policies and execute them.

Complex Gray Release Scenarios

Two illustrative scenarios assume a 1% user gray based on uid modulo.

Scenario 1: Simultaneous Gray of Multiple Services in a Call Chain

The new gateway tags requests with tag T. Downstream services forward requests carrying tag T to their new versions, while requests without the tag go to the old versions.

Scenario 2: Gray Release Involving Data Changes

When new versions introduce additional database fields, data must be duplicated. Writes become dual‑writes to both old and new databases, and an MQ is used to ensure eventual consistency after offline full data copy.

During gray release, data from both databases is compared to verify consistency, ensuring no data loss whether the release succeeds or is rolled back.

Conclusion

The author invites readers to like, share, and follow for more technical content.