Why Alipay Chose MetaQ: Inside Its High‑Performance Distributed Messaging System
Metamorphosis (MetaQ), Alipay’s open‑source Java‑based distributed messaging middleware, offers high throughput, ordered storage, transaction support, and HA replication, positioning itself as a Kafka‑like solution tailored for massive log, broadcast, and sequential data flows, now powering billions of daily messages for Alipay and Taobao.
Alipay’s Open‑Source Distributed Messaging Middleware – MetaQ
Metamorphosis (MetaQ) is a high‑performance, highly available, scalable distributed message middleware developed by Alibaba, similar to LinkedIn’s Kafka, supporting ordered writes, large throughput, local and XA transactions, and suitable for high‑throughput, ordered, broadcast, and log data transmission scenarios.
It is a pure Java solution, unlike Kafka which is written in Scala, and addresses features missing in Kafka such as transaction support, multiple offset storage options, and high‑availability replication.
Why MetaQ was created
Kafka is written in Scala, which the team was unfamiliar with, and its community development was perceived as slow.
Needed functionalities not present in Kafka: transactions, various offset storage mechanisms, and HA replication.
Key Features of MetaQ
Text‑based protocol offering transparent monitoring similar to memcached stats.
Pure Java implementation covering communication, storage, client, and server.
Supports both local and XA distributed transactions.
HA replication with asynchronous and synchronous modes to ensure reliability.
Asynchronous message sending.
Local recovery on consumption failures.
Multiple offset storage options: database, disk, Zookeeper, with customizable group commit.
Message broadcast mode.
Complementary projects: Python client, Twitter Storm spout, Tail4j, etc.
MetaQ processes billions of messages daily: over 120 billion messages are routed through MetaQ each day at Alipay, and Taobao also handles hundreds of millions.
Typical Use Cases
Log transmission – high‑throughput log pipelines.
Message broadcasting – e.g., cache invalidation notifications.
Ordered data synchronization – such as MySQL binlog replication.
Message routing in distributed environments where brokers, producers, and consumers form clusters with strict ordering and reliability requirements.
General purpose MQ scenarios.
Alipay System Architecture Overview
Signed-in readers can open the original source through BestHub's protected redirect.
This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactand we will review it promptly.
21CTO
21CTO (21CTO.com) offers developers community, training, and services, making it your go‑to learning and service platform.
How this landed with the community
Was this worth your time?
0 Comments
Thoughtful readers leave field notes, pushback, and hard-won operational detail here.