Why ZooKeeper Is Essential for Distributed Application Coordination

Goal

ZooKeeper is popular; basic questions: what is it used for and why was it created?

What is ZooKeeper used for?

Why was ZooKeeper created?

Answers: ZooKeeper simplifies distributed application development by hiding low‑level details, exposing a simple API, and providing a high‑performance, highly available, reliable cluster.

In short, ZK solves distributed application development problems.

What common problems do distributed applications face and how does ZK hide them?

What APIs does ZooKeeper expose and how do they support development?

How does ZooKeeper achieve high performance, high availability, and high reliability?

Why ZooKeeper Exists

When multiple processes need to cooperate, business logic becomes complex and reusable. Extracting these common coordination concerns into infrastructure lets developers focus on business logic.

ZooKeeper is such a coordination service.

ZooKeeper Features

API inspired by file‑system APIs, providing simple operations.

Runs on dedicated servers, separate from business logic, ensuring high fault‑tolerance and scalability.

ZooKeeper stores coordination data (metadata), not application data, which should be stored elsewhere (e.g., HDFS). It can be viewed as a special file system.

Application data and metadata have different consistency and durability requirements and should be stored separately.

ZooKeeper Mission

Provide efficient, reliable distributed coordination services such as naming, distributed locks, crash detection, leader election, and configuration management.

Multi‑process Coordination

Two categories: cooperation (processes act together) and competition (mutual exclusion).

Cooperation can be intra‑node (shared memory) or inter‑node (across network). ZooKeeper focuses on the latter.

Cross‑network coordination uses either message mechanisms or shared storage; ZooKeeper adopts the shared‑storage approach.

Common network issues: message delay, processing latency, clock skew. ZooKeeper is designed to hide these three problems.

ZooKeeper Characteristics

Fundamental Problems Solved

Distributed consistency issues: message delay, loss, node crashes.

ZooKeeper uses proposal voting (2PC) and leader election to ensure data consistency.

Paxos aims to solve distributed consistency and improve fault tolerance.

Positioning

ZooKeeper is a distributed coordination service that is high‑performance and reliable, allowing applications to focus on business logic without dealing with low‑level coordination details.

It exposes an API rather than raw primitives, similar to a file‑system API.

Features

Sequential consistency: client requests are processed in order.

Atomicity: transactions are applied to all nodes or none.

Single view (eventual consistency).

Reliability: committed state persists.

Timeliness: guarantees eventual consistency.

Design Goals

High performance: tree‑structured data, all data in memory, followers/observers handle non‑transactional requests.

High availability: majority of nodes must be alive; automatic leader election.

Sequential consistency: all transactions go through the leader and receive a globally unique increasing zxid.

Eventual consistency: proposal voting ensures that once a transaction is committed, a majority sees the update.

Before ZooKeeper

Distributed systems used distributed lock managers or distributed databases for coordination.

ZooKeeper focuses on process coordination without providing lock interfaces or generic storage.

Typical server needs: leader election, crash detection, distributed locks—all addressed by ZooKeeper APIs.

ZooKeeper is not suitable for massive data storage; it stores metadata only.

Glossary

Distributed system: a system spanning multiple physical hosts composed of independent nodes.

Primitive: an indivisible operation such as create, query, delete, etc.