How Ele.me Deployed a Dual‑Active Database System in Just Three Months

Background

Ele.me built a dual‑active (multi‑active) database system to run the same service simultaneously in multiple data‑centers. The project was designed, implemented and launched within three months and required solving data‑consistency, latency, and failure‑handling problems across geographically separated sites.

Challenges & Design Principles

The main technical challenges were:

Traffic splitting and control between same‑city and cross‑region deployments.

Cross‑region latency for data access and synchronization.

Data safety, conflict resolution and consistency under failure.

Derived design principles:

Business Cohesion – keep a complete transaction inside a single data‑center to avoid cross‑region round‑trip latency.

Availability First – prioritize service availability; tolerate temporary inconsistency and repair later.

Correctness Assurance – protect data integrity while maintaining availability.

Business Visibility – make business units aware of their data boundaries so they only operate on local data.

Intelligent DNS‑Based Routing – map users to logical partitions, then to physical data‑centers.

Architecture

An intelligent DNS service deployed in the cloud routes each user request to a specific logical partition based on the user’s POI (point of interest). The partition is then mapped to a physical data‑center, ensuring that most transactions stay within a single zone.

Two zone models are used:

ShardingZone – applications that can be partitioned by region; reads and writes are performed locally, eliminating cross‑region latency. Conflict handling focuses on DRC synchronization and auto‑increment key management.

GlobalZone – for workloads that require strong consistency. Writes occur in a designated primary data‑center while reads may be served remotely, accepting some read latency.

Data replication is performed by the DRC (Data Replication Component). Each data‑center runs a DRC instance that captures change events and pushes them bidirectionally to the peer site.

Switching procedure for a GlobalZone (when the primary data‑center must change):

Issue a BLOCK command to lock writes on the source data‑center.

Complete data synchronization and run verification checks.

Use the Global Zone Switcher (GZS) to reroute user traffic to the target data‑center.

DAL (Data Access Layer) updates the write‑direction so subsequent writes go to the new primary.

Database Refactoring & Operational Tools

Key refactoring steps:

Full‑volume data synchronization for test and production environments.

Introduce millisecond‑level timestamps to resolve write conflicts.

Adjust auto‑increment columns to avoid key collisions across sites.

Migrate databases into the appropriate ShardingZone or GlobalZone topology.

Supporting platforms:

DCP – a data‑validation platform that performs full‑ and incremental checks, structural comparisons and provides repair utilities.

D‑Bus – handles DB/table migration, real‑time incremental sync (including pause/resume), sharding‑to‑single‑table conversion, and data verification.

EMHA – an automated high‑availability framework that coordinates HA switching with DAL and DRC.

DDL operations in a multi‑active environment require careful control of space, latency, locks, timing windows, and risk. Different zone types have distinct DDL strategies. The mm‑ost tool is used to guarantee cross‑region DDL consistency within 3‑5 seconds.

Automation has reduced DBA involvement: over 90 % of DDL releases are self‑served by developers, with DBA intervention required only for exceptional cases.

Benefits

Multi‑active deployment delivers:

Breaking single‑region capacity limits – traffic can be shifted to any data‑center when a site reaches saturation.

Zero‑downtime upgrades – traffic is rerouted before maintenance, allowing online schema changes.

Rapid fault isolation – failures in one data‑center do not affect service in others.

Dynamic traffic balancing based on real‑time load and resource availability.

Future Outlook

Planned extensions include:

Deploying multi‑active zones in public cloud environments with elastic scaling.

Implementing storage‑layer data sharding to further reduce cross‑region data movement.

Automated dynamic scaling of resources according to peak and off‑peak loads.

Researching strong‑consistency mechanisms for multi‑active deployments that require strict data guarantees.