High‑Availability Architecture for a Billion‑User Membership System: ES Dual‑Center Clusters, Traffic Isolation, Redis Caching, and MySQL Migration

The membership system is a core service that connects all business lines; any outage would block order placement across the company, so it must guarantee high performance, high availability, and stable service.

After the merger of two companies, the system needed to unify member data across multiple platforms (mobile apps, mini‑programs, etc.), handling cross‑marketing scenarios such as linking train‑ticket purchases to hotel red‑packets.

Because traffic spikes (e.g., holiday peaks) can exceed 20k TPS, the team designed a dual‑center Elasticsearch (ES) master‑slave cluster: the primary cluster runs in data center A, the standby in data center B, with data synchronized via MQ. Failover is achieved by switching reads/writes to the standby cluster.

To prevent marketing‑driven traffic from affecting the main order flow, a three‑cluster ES architecture was introduced: a dedicated high‑TPS cluster for marketing bursts, isolated from the primary ES cluster serving order‑critical queries.

Deep ES optimizations were applied, including balanced shard distribution, appropriate thread‑pool sizing (cpu * 1.5 + 1), limiting shard memory to ≤50 GB, removing unnecessary text fields, using filters instead of queries, moving sorting to the application layer, and adding routing keys to reduce shard fan‑out.

Redis caching was initially avoided due to real‑time consistency concerns, but after a high‑traffic blind‑box event, a cache layer with a 2‑second distributed lock was added to avoid stale data caused by ES’s near‑real‑time delay.

The underlying relational database (originally SQL Server) could no longer scale beyond a billion rows, so a dual‑center MySQL sharding solution was adopted: >1,000 shards, 1‑master‑3‑slave per data center, with near‑zero replication lag and local‑read routing.

Migration from SQL Server to MySQL employed a phased approach: full data sync, real‑time dual‑write, incremental sync, and gradual traffic gray‑release (A/B testing) while continuously verifying data consistency.

To further improve resilience, the system added a MySQL‑ES master‑slave fallback: if the DAL component or MySQL fails, reads/writes can be switched to ES and later synchronized back.

Additional safeguards include fine‑grained flow‑control (hotspot limiting, per‑caller limits, global TPS caps) and degradation strategies based on response time, error count, and error ratio.

Finally, the article outlines future work on more precise traffic control, account governance, and continuous reliability engineering.