How to Build a 100% High‑Availability Membership System with ES, Redis, and MySQL

ES High Availability Solution

1. ES Dual‑Center Master‑Slave Cluster

After the merger of two companies, the unified membership data exceeds ten billion records, requiring Elasticsearch (ES) to store the unified member relationships. To guarantee ES high availability, a dual‑center master‑slave architecture is deployed across two data centers (A and B). The primary cluster runs in data center A, the backup cluster in B, and data is synchronized via MQ. In case of failure, traffic is switched to the backup cluster, and after recovery, data is synchronized back.

2. ES Traffic Isolation Three‑Cluster Architecture

To protect the primary ES cluster from traffic spikes caused by marketing activities, a separate ES cluster is dedicated to high‑TPS marketing requests, isolating them from the main cluster.

3. Deep ES Cluster Optimizations

Uneven shard distribution causing hot spots.

Thread‑pool size set too high, leading to CPU spikes.

Shard memory allocation exceeding 100 GB.

Redundant text fields doubling storage.

Using query instead of filter, causing unnecessary scoring.

Offloading sorting to the application JVM.

Adding routing keys to limit shard queries.

These optimizations dramatically reduced CPU usage and improved query latency.

Member Redis Cache Scheme

Initially the member system did not use caching because ES performance was sufficient and data consistency was critical. However, a sudden traffic surge during a ticket blind‑box event prompted the introduction of a Redis cache.

1. Solving Redis Inconsistency Caused by ES Near‑Real‑Time Delay

Because ES updates become visible after about one second, a race condition can cause stale data to be written back to Redis. The solution adds a 2‑second distributed lock before deleting the Redis entry, ensuring that queries during the lock do not refresh the cache with stale data.

2. Redis Dual‑Center Multi‑Cluster Architecture

Two Redis clusters are deployed in data centers A and B. Writes are performed to both clusters; reads are served locally, providing high availability even if one data center fails.

High‑Availability Member Primary Database Scheme

1. MySQL Dual‑Center Partition Cluster

The member data (over ten billion rows) is sharded into more than 1,000 partitions, each holding about one million rows. The cluster uses a 1‑master‑3‑slave topology with the master in data center A and slaves in B, synchronized with sub‑millisecond latency.

2. Smooth Migration from SQL Server to MySQL

Migration is performed without downtime using a three‑stage approach: full data sync, real‑time dual‑write, and gradual traffic gray‑release with A/B testing and result verification.

3. MySQL and ES Master‑Slave Cluster

In case of DAL component failures or MySQL outage, the system can switch reads and writes to ES, then synchronize back once MySQL recovers.

Abnormal Member Relationship Governance

Complex logic identifies and fixes cases where a user’s APP account becomes bound to another’s WeChat account, preventing cross‑account data leakage and erroneous order operations.

Future: Fine‑Grained Flow Control and Degradation

1. Fine‑Grained Flow Control

Implements hotspot limiting, per‑caller rules, and global thresholds to protect the system from abusive traffic and unexpected spikes.

2. Fine‑Grained Degradation

Degrades based on average response time and error rates, with automatic circuit‑breaker behavior.