Understanding Eventual Consistency and Anti‑Entropy in Distributed Databases – Part 2

In this blog series we explore eventual consistency, a consistency model used by many distributed systems such as XDB Enterprise, and introduce two essential concepts: hinted handoff queues and anti‑entropy (AE).

The first part ( "Distributed Architecture – Eventual Consistency: Hinted Handoff Queue" ) covered the basics of eventual consistency and its importance in distributed computing.

Part 2

What is anti‑entropy?

If you read the first part you already know how hinted handoff queues preserve data during node outages, but many failure scenarios can still cause data loss. Anti‑entropy (AE) is the second half of maintaining eventual consistency, designed to detect and repair such losses.

Entropy, defined by the second law of thermodynamics, means that ordered systems tend toward disorder over time. Anti‑entropy opposes this trend by eliminating disorder in time‑series data.

AE runs as a service in XDB Enterprise to check for inconsistencies. When a node reports data, AE can identify and fix underlying differences, acting as a hero that restores consistency.

Example 1

Consider a classic cluster with two data nodes and a replication factor of 2 in XDB Enterprise.

The system works well until node 2 experiences hardware failure and goes offline. New writes are queued in the hinted handoff queue (HHQ) until node 2 returns, while reads are served by node 1, which still holds a full copy.

AE detects the missing shards on node 2, copies them from node 1, and flushes the queued writes, bringing both nodes back to the same state.

After the repair, the cluster resumes normal operation with consistent data across both nodes.

Example 2

The HHQ has practical limits: by default it stores up to 10 GB and retains data for 168 hours. Older or excess data is discarded, so HHQ is meant only for short‑term interruption handling.

When a node is down for a long period, the queued data may exceed HHQ limits and be lost. AE mitigates this by continuously checking each node’s shards, copying missing fragments from healthy nodes, and ensuring eventual consistency without manual intervention.

From XDB Enterprise 1.5 onward, AE verifies that every node holds all shards indicated by the meta‑store and automatically replicates missing ones. Starting with version 1.6, AE can also compare shard data for consistency and repair any discrepancies it finds.

In summary, AE acts as a self‑healing service that identifies and fixes missing or inconsistent shards, provided at least one replica exists. It cannot repair hot shards that are actively being written to, and it requires a replication factor of 2 or higher to function effectively.

Summary

Eventual consistency guarantees high availability while keeping data accurate; the combination of HHQ and AE works like a superhero duo that silently resolves data inconsistencies, allowing us to trust our data and focus on business‑critical tasks.