Choosing CP vs AP for Service Discovery: When to Use Zookeeper or a Message Bus

Service Discovery Overview

In high‑availability production environments services are deployed as clusters whose IP addresses can change at any time. Callers need a dynamic "phone book" to obtain the current list of service instances – this process is called service discovery.

1. Core Concepts

The contract between a caller and a provider is an interface name (the "key" in the phone book). The concrete service nodes expose that interface at specific IP/port pairs (the "address list"). A discovery mechanism maps the interface name to the current set of addresses.

2. Service Registration & Subscription

Registration : When a provider starts, it registers the interface name and its IP/port with the registration centre.

Subscription : When a consumer starts, it queries the centre for the list of provider addresses, caches the list locally, and uses it for subsequent RPC calls.

3. Why Not Use DNS?

Mapping an interface to a single domain name and relying on DNS looks simple, but DNS caching (especially the JVM default of an infinite cache) prevents timely removal of failed nodes and delays inclusion of newly added nodes. Even with a load‑balancer (VIP) in front of DNS, the solution adds extra cost, an additional network hop, manual node management, and inflexible load‑balancing strategies, making it unsuitable for most RPC scenarios.

4. Zookeeper‑Based Discovery (CP)

Zookeeper (or etcd) provides strong consistency (CP) and a watch‑based push mechanism.

Create a root znode for each service, e.g. /service/com.example.MyService, with child directories provider and consumer.

When a provider registers, it creates an ephemeral node under provider containing its address and metadata.

When a consumer subscribes, it creates an ephemeral node under consumer and sets a watch on the provider directory.

Zookeeper notifies all watching consumers whenever the set of provider nodes changes.

Drawbacks : During massive roll‑outs, thousands of providers may register simultaneously, causing CPU spikes and even crashes of the Zookeeper ensemble. High read/write frequency and a large number of znodes further degrade stability.

5. Message‑Bus‑Based Discovery (AP)

RPC can tolerate a few seconds of staleness, so we can relax the CP requirement to eventual consistency (AP) for better performance and stability.

Each service registration generates a versioned message and pushes it to a message bus (e.g., Kafka, Pulsar, or any MQ).

The bus forwards the message to all registration‑centre nodes. Each node replays only messages with a version higher than its local version, discarding older ones.

Consumers read the full list of instances for an interface from the registration‑centre’s in‑memory cache.

A push‑pull model delivers incremental updates to consumers, which merge them into their local caches.

This design yields a two‑level cache (registration centre memory + consumer memory) and keeps latency low while guaranteeing eventual consistency.

If a consumer receives a stale address (e.g., the node is already down), the RPC framework validates the target before invoking the method. On failure the request is rejected and the consumer retries another cached address.

6. Comparison and Recommendations

Traditional CP solutions (Zookeeper, etcd) become unstable under massive concurrent registrations because every change must be synchronously replicated to all nodes. An AP‑oriented design using a message bus reduces the load on the registration cluster, tolerates bursty registrations, and still provides timely service updates through versioned messages.

7. Frequently Asked Questions

How are dead nodes removed? Consumers periodically refresh their cache; stale entries are discarded when a registration‑centre receives a newer version without the dead node.

Can traffic weighting be applied? Yes. The registration centre can store weight metadata per instance, and the RPC client can use weighted round‑robin or similar algorithms.

Are there open‑source implementations? Any MQ (Kafka, Pulsar, RocketMQ) can serve as the message bus. Projects such as Apache ServiceComb and Spring Cloud Alibaba have experimental AP‑style registries built on top of a message bus.

How is version ordering guaranteed? A monotonically increasing sequence number or a timestamp generated by the registering node ensures total ordering across the cluster.

Can the same mechanism be reused for configuration distribution? Yes. Incremental, versioned messages can also propagate configuration changes, providing the same eventual‑consistency guarantees.