How a Kafka‑Proxy Boosts Cluster Scalability and Resilience

Challenges of Large Kafka Clusters

As business traffic grows, existing Kafka clusters become massive, with thousands of topics per cluster, leading to stability risks, lack of cross‑cluster disaster recovery, and operational pain points such as cumbersome offset resets.

Design Goals

Seamless cluster switching to isolate core services.

Cross‑center traffic monitoring with detailed metrics and alerts.

Online consumption‑offset reset without downtime.

Topic‑level flow‑control to prevent overload.

Dual‑center support for region‑aware production and consumption.

System Design

The solution adds a lightweight proxy between clients and brokers. The proxy speaks the native Kafka protocol, requiring no changes to existing clients or brokers, and is typically deployed on the same machines as the brokers to minimize latency.

Architecture Overview

Metadata sharing enables multiple backend clusters to be accessed via a single public address. Topics can be moved between clusters transparently, allowing maintenance or upgrades without client awareness.

Internal Components

Netty Server : receives client requests.

Netty Client : forwards requests to brokers.

Key Queue : stores active channel IDs.

DataTable : maps channel IDs to request queues.

SendWorker / Acks0SendWorker : processes normal and acks=0 requests respectively.

ChannelManager : maintains client‑proxy‑broker channel mappings.

Cache Manager : caches cluster, topic, and config metadata.

Processor : applies custom processing before forwarding to brokers.

Workflows

Proxy starts and listens on port 19092.

Incoming requests are parsed to extract ApiKey and acks.

Requests with acks=0 are routed to Acks0SendWorker; others go through SendWorker and appropriate Processor.

Responses are matched to requests via requestId and written back to the client.

On mismatches or exceptions, connections are reset to preserve stability.

Key Features

Seamless Cluster Switching

Topics can be migrated between clusters without client impact; current implementation may lose unconsumed data, with future plans to eliminate loss using Kafka’s native tools.

Near‑Production/Consumption

Traffic is directed to the nearest region/AZ based on client IP, with configurable target clusters and fallback mechanisms.

Online Offset Reset

Administrators can reset consumer offsets via the management platform; the proxy intercepts related requests, returns default responses, and disconnects from brokers to trigger the reset while keeping consumers running.

Flow Control (Production & Consumption)

Topic‑level throttling stops forwarding requests that exceed configured thresholds, returning default responses to protect broker stability.

Benefits

By sharing metadata and inserting the proxy, the organization split a massive Kafka cluster into multiple dedicated clusters, achieving higher resource utilization, improved fault isolation, detailed traffic monitoring, and reduced network overhead through batch and compression tuning.

Future Outlook

Automated dual‑center failover scripts.

Expanded testing across client versions.

Seamless Kafka version upgrades via the proxy.

These enhancements aim to further increase reliability, operational efficiency, and flexibility for large‑scale messaging workloads.