How Kafka Clients Choose the Correct Listener in Multi‑Network Environments

When a Kafka cluster configures multiple listeners per broker, a client must decide which listener address to use for its requests. The article starts with a colleague's question and sets up the background that each broker can define several listeners and that the client only needs to specify a subset of broker addresses via bootstrap.servers.

Metadata Retrieval Process

The client (producer or consumer) obtains cluster metadata through the MetadataUpdater . After the broker returns a MetadataResponse, the client parses it in Metadata.handleMetadataResponse. The relevant snippet shows how a full or partial update builds a MetadataCache containing a map of broker IDs to Node objects.

private MetadataCache handleMetadataResponse(MetadataResponse metadataResponse, boolean isPartialUpdate, long nowMs) {
    // ...
    Map<Integer, Node> nodes = metadataResponse.brokersById();
    if (isPartialUpdate)
        return this.cache.mergeWith(metadataResponse.clusterId(), nodes, partitions, ...);
    else
        return new MetadataCache(metadataResponse.clusterId(), nodes, partitions, ...);
}

Debug Example

A three‑broker cluster is started with the following listener configurations:

server0: listeners = PLAINTEXT://localhost:9090 server1: listeners = PLAINTEXT://localhost:9091,TEXT://localhost:9099 server2: listeners = PLAINTEXT://localhost:9090 A KafkaProducer is launched with bootstrap.servers=localhost:9099. Debugging shows that the metadata cache contains only the endpoint localhost:9099 (id: 1 rack: null). This indicates that the broker has already filtered the endpoint list based on the listener that accepted the client connection.

Server‑Side Network Model

During broker startup, an Acceptor and a Processor are created for each configured listener. The ChannelBuilder associated with an Acceptor stores the listener name. Consequently, every request arriving at a Processor carries the originating listenerName.

Request Handling Flow

When a request is received, the Processor assembles a Request object that includes the listenerName. In the handling of handleTopicMetadataRequest, the broker filters the list of broker endpoints using this name:

brokers.flatMap(_.getNode(request.context.listenerName)).asJava,

Thus only endpoints whose listener name matches the one used by the client are returned.

Conclusion

The client’s choice of listener is determined by the address specified in bootstrap.servers. The broker then filters the metadata to include only endpoints that share the same listener name, ensuring that subsequent requests are routed through the same listener.

Implications of Mixed Listener Names

If the bootstrap.servers list contains addresses that belong to different listener names, the metadata returned can vary depending on which address the client contacts first. The article presents a scenario with three brokers, each exposing three listeners (PLAINTEXT1, PLAINTEXT2, PLAINTEXT3). When the client initially contacts IP1:9090 (listener PLAINTEXT1), it receives all brokers with that listener. Later, contacting IP3:9091 (listener PLAINTEXT2) yields a reduced set containing only brokers that expose PLAINTEXT2. Therefore, to avoid inconsistent metadata, all addresses in bootstrap.servers should belong to the same listener name.