Zookeeper vs etcd: Which Distributed Coordination System Fits Your Needs?

Zookeeper and etcd are both highly regarded distributed coordination systems; Zookeeper originates from the Hadoop ecosystem, while etcd gained popularity as the backing store for Kubernetes.

Zookeeper

Overview

Zookeeper started in Hadoop and evolved into a top‑level Apache project, now widely used in Apache projects such as Hadoop, Kafka, and Solr.

Zookeeper uses the ZAB protocol for consistency and operates with an odd number of nodes forming a quorum. One node becomes the leader after election, handling all write requests, while any node can serve read requests. This design ensures CP guarantees in the CAP model.

ZNode

Zookeeper stores data in a hierarchical tree of ZNodes, each addressed by an absolute path from the root. A ZNode can hold up to 1 MB of data, and clients can create, delete, set data, and read data from ZNodes.

Watches

Clients can set a watch on a ZNode to receive a one‑time notification when the node changes (creation, deletion, data update, or child changes). After a notification, the watch must be re‑registered.

Advantages

Non‑blocking snapshots achieving eventual consistency

Efficient memory management

High reliability

Simple API

Automatic connection retries

Well‑tested ZooKeeper recipes

Easy to implement new recipes

Event watching support

During network partitions, the minority partition stops, preserving consistency

Disadvantages

Implemented in Java, inheriting Java GC pauses

Snapshot writes pause read/write operations temporarily

Each watch opens a new socket, leading to complex socket management

etcd

Overview

etcd is written in Go and, although newer than Zookeeper, has strong prospects due to its role in Kubernetes, where the kube‑master uses etcd for distributed storage and locking.

etcd implements consistency with the Raft algorithm, which is simpler than Zookeeper’s ZAB. It provides a distributed key‑value store rather than a hierarchical tree.

API

etcd3 offers the following operations:

put – add a new key‑value pair

get – retrieve the value of a key

range – retrieve values for a range of keys

transaction – combine read, compare, modify, and write

watch – monitor a key or range for changes

Advantages

Supports incremental snapshots, avoiding pause issues

Off‑heap storage eliminates GC pause problems

Watch connections are reusable, reducing socket overhead

Watches are continuous; no need to re‑register after a trigger

etcd3 retains events in a window, preventing loss after client disconnect

Disadvantages

If a request times out or the client loses network connectivity, the client may not know the operation status

During leader election, etcd aborts operations without sending an explicit abort response

In a network partition, reads may still be served by a leader in a minority partition

Conclusion

Zookeeper, written in Java, is widely adopted by many Apache projects and is known for its stability.

etcd, written in Go, is primarily used by Kubernetes and is gaining traction.

Zookeeper offers mature client libraries for many languages, while etcd’s Go‑centric ecosystem may require HTTP calls for non‑Go clients.

Choosing between Zookeeper and etcd depends on your specific requirements, consistency guarantees, performance needs, and the programming language of your application.