Stability Governance of Xianyu Messaging System

Introduction

Xianyu, a C2C e‑commerce platform, relies on its messaging system to build trust between buyers and sellers. System stability directly impacts user experience and transaction efficiency. In August 2022 the team launched a systematic stability‑governance program.

Problem Definition

The goal is to reduce online incidents. Issues were classified into high‑risk/high‑probability problems (e.g., change risk, weak dependency risk) and deep‑water problems with high remediation cost (e.g., strong dependency risk, architectural flaws). Corresponding measures include gray‑release, monitoring & alerts, automated regression, dependency management, drills, and refactoring.

Problem Governance

Gray Release

A “safe‑production” environment receives 1% live traffic plus full internal traffic, providing a closed‑loop for validation. MQ topics were isolated via Spring Conditional beans to keep traffic within the safe environment.

Monitoring Alerts

Separate monitoring dashboards, alert thresholds, and offline reports were created for the safe‑production environment, covering request volume, latency, error rates, and message delay. Continuous review of coverage, timeliness, and effectiveness ensures long‑term alert health.

Automated Regression

End‑to‑end regression tests are integrated into CI/CD and run daily. Interface‑level traffic replay using the Phoenix tool (built on JVMTI) records and replays RPC traffic to verify stability.

Dependency Governance

Dependencies are reviewed at code level, unnecessary strong dependencies are downgraded to weak ones, and dedicated monitoring and rapid‑recovery plans are added. Dependency drills validate the expected behavior of strong and weak links.

Conclusion

Six months after implementation, online incidents have approached zero. The experience shows that stability governance requires focused, context‑specific measures, continuous investment, and a vigilant mindset toward every change.