Design and Architecture of MQProxy: A Distributed Message Queue Proxy for Kafka

Background : Different development teams in the department used various languages and had to manage Kafka selection, protocols, and health monitoring themselves, leading to stability and fault‑tolerance issues. The MQ team created MQProxy, a Java client‑based proxy that provides a simple SDK for production and consumption.

Design Goals : Enable developers to use message‑queue services without worrying about underlying system selection, protocols, usage patterns, health status, or fault‑tolerance mechanisms.

Implemented Core Functions :

Support for common open‑source MQ systems.

Unified access via HTTP/TCP protocols.

Simple produce/consume/commit APIs.

Health monitoring with built‑in fault‑tolerance and reliability.

Advanced Features include delayed queues, dead‑letter queues, message audit/trace, timeout and proactive retries, message callbacks, unlimited consumer coordination, multi‑datacenter active‑active consumption, and multi‑language SDKs (PHP, Go, Java).

Logical Architecture : MQProxy uses Apache Kafka as the default backend (with future support for RocketMQ, Pulsar, etc.) and consists of three main modules – ProducerProxy, ConsumerProxy, and a Delay‑Queue module. The overall flow involves HTTP communication through an Nginx gateway, local storage via MapDB, and metadata persistence in MySQL and ClickHouse.

ProducerProxy Design : Provides Kafka disaster‑recovery by switching to a backup Kafka when the primary fails, stores messages locally before sending, and supports horizontal scaling up to ~20,000 QPS per node.

ConsumerProxy Design : Built on Dubbo for high‑throughput, stateless service; uses a multi‑step process involving pre‑fetching from Kafka, portal registration in Zookeeper, offset submission, and a dedicated offset‑merge module that consolidates offsets via a log‑compacted Kafka topic.

Delay‑Queue Proxy : Implements both fixed‑level and arbitrary‑level delayed messaging by routing messages through a hierarchy of delay topics and using Java's DelayQueue for countdown timers.

Future Plans : Consolidate MQProxy clusters into a shared resource pool, manage consumers at the partition level, switch to low‑level consumer APIs with Helix for better rebalance handling, add long‑connection support for ordering and transactions, and expand multi‑language SDK coverage.