Understanding Eventual Consistency and Anti‑Entropy in Distributed Systems

In this blog series we explore eventual consistency, a consistency model used by many distributed systems such as XDB Enterprise, and the two key concepts that support it: hinted handoff queues and anti‑entropy (AE).

What is anti‑entropy? Entropy, defined by the second law of thermodynamics, means that over time ordered systems tend toward disorder. Anti‑entropy is the process of fighting that disorder in time‑series data, ensuring that data replicas converge to the same state.

AE is a service that runs in XDB Enterprise to detect and repair inconsistencies. When a node goes offline, writes are queued in the hinted handoff queue (HHQ) until the node returns, while reads are directed to the remaining healthy replica.

Example 1 shows a cluster with two data nodes and a replication factor of 2. If node 2 fails, writes are stored in HHQ and reads go to node 1. When node 2 is replaced with new hardware, AE checks the shard distribution, copies missing shards from node 1 to node 2, and flushes the queued writes, restoring full consistency.

Example 2 highlights HHQ limits: by default HHQ holds up to 10 GB and retains data for 168 hours. If these limits are exceeded, older data is discarded, so HHQ is meant only for short‑term disruption handling.

From XDB Enterprise 1.5 onward, AE scans each node for missing shards and copies them from healthy peers. Starting with version 1.6, AE can also verify shard data consistency across nodes and automatically repair any mismatches, achieving self‑healing eventual consistency.

AE requires at least one replica of each shard to be available; with a replication factor of 2 this works well, but with RF = 1 the system lacks a truth source for repairs. AE does not handle hot shards (actively written) because their data changes continuously, making accurate checksum comparison impossible.

Summary – Eventual consistency provides high availability while guaranteeing data accuracy over time. The combination of HHQ and AE works like a superhero duo, silently correcting inconsistencies in the background so that applications can trust their data without manual intervention.