Cross-Data-Center Multi-Region Redis Synchronization Using CRDT for Strong Eventual Consistency
This article examines Ctrip's approach to achieving multi‑region, bidirectional Redis replication by employing CRDT techniques and strong eventual consistency to overcome the limitations of single‑directional sync, network partitions, and data‑conflict challenges in distributed database systems.
Redis cross‑data‑center (DC) synchronization is essential for enterprise disaster recovery and global service deployment. Ctrip initially implemented one‑way replication from Shanghai to overseas sites with ~180 ms latency, but business requirements evolved to need independent read/write capabilities at each site and multi‑directional data consistency.
The team evaluated CAP theory for multi‑active regions, prioritizing Partition tolerance (P) and Availability, while accepting weaker Consistency (C). They selected a "strong eventual consistency" model to balance these trade‑offs.
To address consistency, the solution adopts Conflict‑Free Replicated Data Types (CRDT). Two replication styles are discussed: state‑based (full state transfer) and operation‑based (incremental operations). The system combines both: op‑based for normal traffic to reduce bandwidth, and state‑based for recovery when resources are insufficient.
For Redis String operations, the CRDT Register abstraction is used, with two concrete implementations: Multi‑Value (MV) Register for integer increments and Last‑Write‑Wins (LWW) Register for SET commands. The final design integrates an OR‑Set (Observed‑Remove Set) with LWW Register to realize Redis K/V semantics.
Key data structures are illustrated below:
struct CRDT.Register {
string key;
string val;
TAG delete-tag;
int timestamp;
}A Java implementation of an LWW Register is provided, showing how commands are processed, clocks merged, and values updated:
public class LWWRegister extends AbstractCrdt
{
private T value;
private StrictVectorClock clock;
public LWWRegister(String nodeId, String crdtId) {
super(nodeId, crdtId, BehaviorProcessor.create());
this.clock = new StrictVectorClock(nodeId);
}
protected Option
> processCommand(SetCommand
command) {
if (clock.compareTo(command.getClock()) < 0) {
clock = clock.merge(command.getClock());
doSet(command.getValue());
return Option.of(command);
}
return Option.none();
}
// ... other methods omitted for brevity
}An OR‑Set implementation is also shown, handling add/remove operations with tombstones to resolve concurrent deletions.
public class ORSet
extends AbstractSet implements Crdt
> {
private final Set
> elements = new HashSet<>();
private final Set
> tombstone = new HashSet<>();
// add, remove, prepareAdd, prepareRemove, etc.
}The article concludes that CRDT provides a practical way to achieve multi‑region, bidirectional Redis synchronization with strong eventual consistency, and invites readers to follow future technical posts or the open‑source XPipe project for deeper details.
Ctrip Technology
Official Ctrip Technology account, sharing and discussing growth.
How this landed with the community
Was this worth your time?
0 Comments
Thoughtful readers leave field notes, pushback, and hard-won operational detail here.