How Xianyu’s Messaging Team Built a Zero‑Incident System with Gray Releases, Monitoring, and Automated Regression

Background

The Xianyu C2C marketplace relies on a real‑time messaging system to establish buyer‑seller trust. Instability in this system directly degrades user experience and transaction efficiency. In August 2022 the team launched a systematic stability‑governance program aimed at reducing online incidents.

Problem Classification

Historical incident analysis identified two categories of risk:

High‑risk, high‑probability issues – e.g., change‑induced regressions and weak‑dependency failures.

Deep‑water issues – e.g., strong‑dependency bottlenecks and architectural flaws that require long‑term investment.

The governance effort focused on the first category using gray releases, fine‑grained monitoring, automated regression, dependency governance, and operational drills.

Gray Release (Safe‑Production Environment)

A dedicated “safe‑production” environment receives 1 % of live traffic plus 100 % of internal traffic. Traffic is routed through a gateway that isolates the environment from the main production cluster.

To prevent MQ‑based load‑balancing from leaking traffic, the team used Spring’s @Conditional bean injection to provide separate MQ topics for the safe‑production and production environments. This guarantees that all messages stay within the safe‑production loop.

Key artifacts:

Independent monitoring dashboard – tracks call volume, latency, error rate, and message delay for core flows.

Separate alert thresholds – thresholds are tuned for the safe‑production baseline.

Monitoring & Alert Governance

The alert lifecycle consists of data preparation, configuration, verification, rule definition, and validation. Governance focuses on three dimensions:

Coverage – Identify core scenario chains, enumerate missing metrics, and add corresponding alerts.

Fallback alerts – Generic alerts for resource usage, interface latency, and middleware health.

Review – Offline monitoring reports aggregate alert history and support periodic coverage reviews.

Timeliness and effectiveness are balanced by tightening alert conditions incrementally and re‑evaluating them in each review cycle.

Automated Regression

Regression tests are triggered automatically by the CI/CD pipeline whenever a build is deployed to the safe‑production environment.

End‑to‑end scenarios – Tests cover installation, usage, and uninstallation of the messaging client, exercising all critical paths.

Interface‑level traffic replay – The Phoenix replay tool (implemented with JVMTI) records RPC traffic, replays it against the new build, and diffs the results to detect regressions.

Release Specification for Safe‑Production

After a build passes safe‑production validation, the release process requires:

Retain the build in safe‑production for at least one night to capture time‑related bugs.

Perform a gray rollout the following day, gradually increasing traffic while monitoring the dedicated dashboards.

Generate a t+1 offline monitoring report to confirm that key metrics remain within acceptable ranges.

Dependency Governance

The goal is to keep strong‑weak dependency relationships reasonable and ensure that weak dependencies can degrade and recover quickly.

Dependency audit – Code‑level review of each dependency’s rationale and its fast‑recovery capability.

Refactoring – Convert unsuitable strong dependencies into weak ones, add monitoring for the new weak paths, and define degradation plans.

Drills – Simulate failure of weak dependencies to verify that the system detects the issue and recovers as expected.

Outcomes

Within six months of implementing the above practices, online incidents fell to near zero. The team observed that stability work is continuous, must target high‑impact areas, and requires ongoing investment such as alert tuning and dependency drills.

Problems never disappear completely; the aim is reduction, not elimination.

Governance must be tailored to real‑world constraints; generic solutions may not fit.

Alert configurations need regular review to stay effective.

Every production change should be treated with caution and verified in the safe‑production loop.