High‑Availability Architecture and Optimization Strategies for a Large‑Scale Membership System

Background – The membership system is a core service for all business lines; any outage blocks order placement across the company. After the merger of Tongcheng and eLong, the system must support cross‑platform member queries for multiple apps and mini‑programs, handling traffic spikes exceeding 20,000 TPS during holidays.

ES High‑Availability Solution – A dual‑center primary‑backup Elasticsearch cluster is deployed across two data centers (A and B). The primary cluster handles reads/writes, while changes are replicated to the backup via MQ. In case of node or data‑center failure, traffic is switched to the backup cluster within seconds, and later synchronized back.

ES Traffic Isolation Three‑Cluster Architecture – To protect the main ES cluster from massive marketing‑driven traffic, a separate ES cluster handles high‑TPS flash‑sale requests, ensuring the order‑critical path remains unaffected.

Deep ES Optimizations – Issues such as uneven shard distribution, oversized thread pools, large shard memory (100 GB), redundant text/keyword fields, misuse of query instead of filter, and lack of routing keys were addressed. After tuning, CPU usage and query latency dropped dramatically, as shown in the performance charts.

Member Redis Cache Scheme – Initially avoided caching due to real‑time consistency concerns, but introduced a Redis cache with a 2‑second distributed lock to prevent stale data after Elasticsearch’s near‑real‑time delay. Cache hit rates exceed 90 %, greatly reducing ES load.

Redis Dual‑Center Multi‑Cluster Architecture – Two Redis clusters (one in each data center) are written synchronously; reads are served locally, providing high availability even if an entire data center fails.

High‑Availability Member Primary Database – The relational member data migrated from a single SQL Server instance to a dual‑center MySQL partitioned cluster (over 1,000 shards, 1‑master‑3‑slave setup). Real‑time dual‑write ensures seamless cut‑over, with automated gray‑scale traffic migration and consistency checks.

Migration and Gray‑Scale Strategy – A phased approach of full data sync, real‑time dual‑write, incremental sync, and A/B traffic gray‑scale was used to move from SQL Server to MySQL without downtime, with fallback mechanisms for any inconsistency.

Abnormal Member Relationship Governance – Complex logic was added to detect and block erroneous cross‑account bindings, preventing data leakage and financial loss.

Future Fine‑Grained Flow‑Control and Degradation – Plans include hotspot throttling, per‑client flow‑control, global rate limiting, and degradation based on average response time or error ratios to ensure the system remains resilient under extreme load.