Service Discovery: Comparing Zookeeper, etcd, Consul and Related Tools

Service Discovery Tools

When the number of services grows, using predefined ports leads to conflicts; Docker can assign random ports, but we must discover and share those ports with other services.

In distributed systems deployed across multiple servers, manually assigning services to specific hosts becomes impractical, hindering auto‑scaling and fault‑tolerant recovery. Therefore, we need processes for service registration (storing host and port) and service discovery (allowing others to locate registered services).

Additional concerns include deregistering stopped services, handling multiple instances, load balancing, and node failures, all of which are tightly coupled with registration and discovery mechanisms that typically rely on highly available distributed key/value stores.

Manual Configuration

Manually managing service placement, configuration, and scaling is error‑prone and does not scale; adding a second instance requires repeating the entire manual process, and failure recovery is slow.

Zookeeper

Zookeeper, originating from the Hadoop ecosystem, provides a mature, reliable distributed configuration store with a filesystem‑like hierarchy and leader election. Its drawbacks are heavy Java dependencies, complexity, and resource consumption.

etcd

etcd is an HTTP‑accessible distributed key/value store that offers simple deployment, strong data persistence, security, and good documentation. It is easier to use than Zookeeper but often needs additional tools to achieve full service‑discovery functionality.

Registrator

Registrator watches Docker containers and automatically registers or deregisters services in etcd, Consul, or SkyDNS 2 when containers start or stop.

confd

confd is a lightweight configuration management tool that pulls data from etcd, Consul, or other registries to keep configuration files up‑to‑date and can reload applications when those files change.

Combining etcd, Registrator, and confd

Together they provide a simple yet powerful solution for automating service discovery and configuration without introducing unnecessary complexity.

Consul

Consul is a robust distributed key/value store with built‑in service discovery, health checks, multi‑datacenter support, and a DNS/HTTP API. Unlike Zookeeper and etcd, Consul includes an embedded service‑discovery framework, eliminating the need for third‑party tools.

Consul’s health‑check capabilities, web UI, and support for multiple data centers make it a comprehensive solution for monitoring and managing services.

Consul Template and Consul‑Registrator Integration

Registrator supports both Consul KV and Consul protocols, storing additional metadata such as node information, service ID, and tags. Consul Template (similar to confd) can render configuration files from Consul data and execute arbitrary commands on updates.

Final Thoughts

All tools share similar principles: they run on nodes, require consensus, and provide a key/value store. Zookeeper is the oldest and most complex; etcd paired with Registrator and confd offers a simple, powerful combination; Consul adds built‑in service discovery, health checks, and multi‑datacenter support, often making it the most complete solution.

Choosing the right tool depends on the specific requirements and the balance between simplicity and feature richness.