Why ZooKeeper Fails as Service Discovery: Alibaba’s 10‑Year Lessons

Is ZooKeeper Really the Best Choice for Service Discovery?

Looking back at the history of service discovery, one may wonder what would happen if ZooKeeper had been introduced earlier than Alibaba’s own ConfigServer. In practice, ZooKeeper cannot be considered the optimal solution for a registration center.

Requirements Analysis and Key Design Considerations for a Registry

The core function of a registry is a query operation: Si = F(service-name), where service-name is the query key and the returned value is the list of endpoints (ip:port).

Note: the term service will be abbreviated as svc in the following text.

Inconsistent endpoint lists (CAP’s C not satisfied) can cause traffic imbalance among service instances, but as long as the registry converges to a consistent state within the SLA (e.g., 1 s), the impact is acceptable.

When a service with 10 replicas returns two different endpoint sets, the traffic will be slightly unbalanced, but the system can quickly converge to a statistically even distribution.

In the CAP trade‑off, a registry should prioritize availability (AP) over strong consistency (CP). Data inconsistency is tolerable, while sacrificing availability breaks the fundamental rule that a registry must never disrupt service connectivity.

Service Scale, Capacity, and Connectivity

As the number of micro‑services and instances grows, ZooKeeper quickly becomes a bottleneck because its write path does not scale horizontally. While ZooKeeper works well for coarse‑grained coordination (locks, leader election) in big‑data workloads, it struggles with high‑frequency registration and health‑check writes required by large‑scale service discovery.

Increasing service scale leads to massive write pressure on the registry; adding more ZooKeeper nodes does not solve the fundamental write scalability limitation.

Does a Registry Need Persistent Storage and Transaction Logs?

ZooKeeper’s ZAB protocol writes a transaction log and periodic snapshots to guarantee durability, which is valuable for coordination data but unnecessary for the real‑time address lists used in service discovery. Persistent storage is only needed for metadata such as version, group, data‑center, weight, and auth policies.

Service Health Check

When ZooKeeper is used as a registry, health checking often relies on session activity and Ephemeral ZNodes, effectively tying health to TCP connection liveness. This approach is insufficient because a healthy TCP session does not guarantee that the service itself is healthy. Registries should provide richer, pluggable health‑check mechanisms that let services define their own health logic.

Disaster‑Recovery Considerations for the Registry

The registry must not become a single point of failure. Service calls should be weakly dependent on the registry, using it only for registration, scaling, or topology changes. Clients should cache registry data (client snapshot) and handle complete registry outages gracefully.

ZooKeeper’s native client lacks built‑in cache and graceful degradation; therefore, when all ZooKeeper nodes go down, production services must still operate without impact.

Expertise Required for Using ZooKeeper

Understanding ZooKeeper’s client/session state machine, handling exceptions such as ConnectionLossException and SessionExpiredException, and dealing with network partitions are non‑trivial. Mis‑handling these scenarios can lead to lost events, duplicate creations, or stale locks.

Developers must decide whether a request is idempotent and choose appropriate retry semantics when connections flicker.

Where ZooKeeper Fits

Alibaba maintains a large ZooKeeper cluster (nearly a thousand nodes) and a production‑grade fork called TaoKeeper. ZooKeeper excels in coarse‑grained coordination for big‑data workloads but is ill‑suited for high‑throughput service discovery and health monitoring in transaction‑critical systems.

Thus, for service discovery, consider alternatives (e.g., Eureka, Consul, Nacos) that prioritize availability and scalability over strict consistency.

Conclusion

The goal of this article is not to denounce ZooKeeper entirely but to share Alibaba’s ten‑year production experience with service discovery, highlighting pitfalls and design lessons that can help the community build better registration centers.