How Ctrip Built Hermes: A Deep Dive into Scalable Message Queue Architecture

Message Queue Advantages

Message queues decouple producers and consumers, enable asynchronous processing, absorb traffic spikes, and support fan‑out scenarios, which are essential for real‑time personalization and high‑throughput workloads in large enterprises.

Evolution of Ctrip’s MQ Architecture

Version 1.0 – MongoDB‑Based Queue

The initial design stored messages directly in MongoDB collections without a broker. While development cost was low, scaling required client upgrades, coordination relied on DB‑level find‑and‑modify, and features such as consumer groups were unavailable.

Version 2.0 – Introducing a Broker

A broker layer was added while still persisting messages in MongoDB. The broker operated in a master‑slave mode, using MongoDB heartbeats for liveness. This enabled consumer groups and centralized coordination, reducing client‑side complexity.

Current Architecture – Hybrid MySQL/Kafka Backend

The latest design adds a meta‑server for cluster coordination and splits storage between MySQL (for critical data) and Kafka (for high‑throughput, less critical streams). Brokers now communicate with the meta‑server via ZooKeeper and lease‑based locks.

Message Types: Kafka vs MySQL

Kafka offers ultra‑high throughput by writing to memory and relying on OS‑level flushing, but lacks built‑in features such as message replay, priority, and fine‑grained filtering. For mission‑critical streams, Ctrip stores messages in MySQL, allowing full control over reliability, monitoring, and custom queue features.

Building an Efficient MQ

Single‑Node Optimizations

Design tables with only an ID primary key and minimal indexes to keep inserts fast.

Use partitioned tables (by ID or date) and drop partitions for cheap data cleanup.

Batch inserts (e.g., rewriteBatchedStatements=true) can yield 5× performance gains.

Implement event‑driven polling to avoid constant DB scans.

Cache recent messages in memory to reduce DB read latency.

Scaling from Single Node to Cluster

Adding brokers behind a load balancer increases capacity but introduces ordering challenges. Topic partitioning solves this: each partition is handled by a single broker, preserving order within the partition while allowing parallel processing across partitions.

Lease‑Based Cluster Management

The meta‑server issues time‑bound leases to brokers and consumers, acting as a lightweight lock. Brokers renew leases to retain ownership of partitions; if a lease expires, the meta‑server rebalances the partition to another broker. Consumers obtain leases to claim partitions, enabling dynamic load‑balancing without ZooKeeper coordination for consumers.

Meta‑server HA is achieved via DNS round‑robin and internal leader election using ZooKeeper; clients query a stable domain name, receive any meta‑server instance, and are redirected to the current leader if necessary.

Conclusion

Key takeaways for building a high‑performance MQ include: keep the write path lightweight with batch inserts and minimal indexing; use partition‑sticky routing to preserve order while scaling; employ a lease‑based meta‑server for simple, reliable cluster coordination; and combine MySQL for critical streams with Kafka for massive, less‑critical workloads.