Design and Practices of Qunar's Self‑Developed High‑Availability Message Middleware

Qunar's infrastructure team introduced the design of their self‑developed message queue middleware, which underpins all transaction flows such as order and payment processing. The system was built in 2012 to address the need for reliable message‑driven architecture after splitting large monolithic services.

The middleware follows a simple three‑component model—producer, broker, and consumer—where each message is the smallest unit, enabling easy scaling via hash‑based partitioning. To guarantee consistency, the team uses a single‑transaction approach: order data and the corresponding message are persisted in the same database transaction, and the message is only sent after the transaction commits.

When a message fails to send, a background task retries delivery, ensuring that a successful business operation always results in a dispatched message. This design prioritizes reliability and stability over raw performance, acknowledging the high cost of message loss in e‑commerce.

Fault tolerance is achieved through multi‑region broker clusters, automatic failover to alternate data centers, and priority routing that prefers the local cluster but falls back when it becomes unavailable. The broker also supports dynamic quota enforcement and throttling to protect the system from abusive subjects.

Isolation is handled by assigning per‑subject quotas and dedicated thread pools or actor‑based dispatchers, preventing a single hot subject from exhausting resources. Governance features include dynamic quota adjustment, reliability levels, selective message replay, and detailed logging for troubleshooting.

Operational tools provide visual tracing of message flow, back‑tracking, replay, and monitoring of QPS and latency at the subject and consumer granularity. Full‑link tracing integrates with QTracer to follow a message across services.

On the consumer side, the team implemented health‑check based online/offline control, idempotency mechanisms (using Redis or MySQL), and best‑effort ordering guarantees. They also discussed strategies for maintaining order, such as state‑machine validation and version‑based filtering.

The article concludes with a Q&A covering topics like message deduplication, cross‑region disaster recovery, idempotency requirements, storage implementation (MySQL + Java), and recent architectural refinements.