This article dissects common MQ mis‑selections, categorises four message types, explains the five architectural roles of a message broker, compares Kafka, RabbitMQ and Pulsar on capabilities and trade‑offs, and provides a step‑by‑step selection guide with real‑world code snippets and best‑practice patterns.
Kafka and Pulsar are both mature distributed messaging platforms, but they differ fundamentally in architecture, performance, scalability, and operational complexity; this article analyzes their core designs, benchmarks, engineering trade‑offs, and real‑world scenarios to guide architects in making a reliable technology selection.
This article walks through the interview question of designing a message queue that handles billions of messages daily and peaks at millions of QPS, covering traffic calculations, core roles, storage and throughput techniques, scalability, high availability, observability, framework comparisons, a real‑world case study, and key follow‑up interview topics.
This article examines the core evaluation criteria for enterprise message queues and provides a detailed comparison of Kafka, RabbitMQ, and Pulsar—including performance, reliability, scalability, operational complexity, ecosystem maturity, and business fit—offering actionable guidance for selecting the optimal solution in cloud‑native environments.
This guide walks through Pulsar's operational challenges and demonstrates how to use Ansible Playbooks—including syntax, modules, roles, variables, and task dependencies—to fully automate the deployment of Zookeeper and Pulsar clusters, dramatically reducing manual effort and configuration errors.
During a gray‑scale upgrade, a Vivo Pulsar deployment using the KoP protocol experienced a sudden drop in partition consumption rates because the consumer group name vanished from Zookeeper nodes, prompting an in‑depth analysis of the root cause, replication steps, and three progressively refined remediation strategies.
This article details Vivo's end‑to‑end Pulsar observability solution, covering the challenges of Prometheus‑based monitoring, the architecture of the alerting pipeline, adaptor development, metric optimizations for subscription backlog and bundle load, and fixes for kop lag reporting issues.
This article explains the three message delivery semantics, the common causes of duplicate messages in queue systems, and presents concrete producer‑side, broker‑side, and consumer‑side deduplication techniques for Kafka and Pulsar, including code samples and best‑practice recommendations.
This article explains why high‑throughput producers and consumers can saturate CPU, memory, network and disk I/O in TDMQ Pulsar clusters, describes the built‑in cluster‑level distributed and topic‑partition rate‑limiting mechanisms, and provides practical guidance for configuration, monitoring, and troubleshooting.
Message queues can deliver duplicates under at‑least‑once semantics, so to protect idempotent business logic such as orders and payments you should combine producer‑side idempotence (e.g., Kafka’s enable.idempotence), broker‑side deduplication (e.g., Pulsar), and a consumer‑side guard using unique IDs stored in a database or Redis.
Message queues can cause duplicate messages that affect idempotent business processes, so this article explains the three delivery semantics (At Least Once, Exactly Once, At Most Once), the causes of duplication, and practical deduplication techniques for producers, brokers (Kafka, Pulsar), and consumers using code examples.
This comprehensive guide explains the role of message queues in distributed systems, compares the architectures and core concepts of Kafka, Pulsar, RocketMQ, RabbitMQ, and NSQ, and provides practical criteria for selecting, configuring, and operating a message‑queue solution in high‑performance, high‑availability environments.
This comprehensive guide compares five popular message queue systems—Kafka, Pulsar, RabbitMQ, RocketMQ, and NSQ—covering their architectures, core concepts, selection criteria, feature differences such as ordering, latency, durability, scaling, and operational considerations, to help engineers select the most suitable solution for their use cases.
The article surveys major message‑queue systems—Kafka, Pulsar, RocketMQ, RabbitMQ, and NSQ—detailing their architectures, key features, selection criteria such as ordering, scalability and latency, performance benchmarks, operational considerations, and typical use cases, and advises which platform best fits high‑throughput, reliability, low‑latency, or lightweight Go‑centric scenarios.
This article explains Apache Pulsar’s fast‑growing adoption, its compute‑storage separation architecture, BookKeeper‑based persistence, multi‑tenant support, flexible subscription models, and fault‑tolerant design, providing a comprehensive overview for developers interested in modern distributed messaging middleware.
Tencent Cloud TDMQ for Apache Pulsar achieves multi‑AZ high availability by containerizing ZooKeeper, BookKeeper and Brokers, using managed ZK, persistent cloud disks and elastic NICs, enforcing quorum and rack‑aware replicas, and planning cross‑region bidirectional replication to ensure seamless disaster recovery and continuous messaging.
This article dissects Pulsar’s storage layer by examining the Bookie component, detailing its architecture, the sequential write‑ahead log, ledger management, journal handling, caching mechanisms, and the complete read/write call chains with concrete code examples and performance considerations.
This technical presentation details Vivo's end‑to‑end big‑data architecture, the evolution from Kafka to Pulsar for massive message processing, deployment strategies, high‑availability mechanisms, observability practices, and future plans for cloud‑native, containerized messaging middleware.
The article walks through building a custom OpenTelemetry Java‑agent extension for Pulsar client metrics—migrating from SkyWalking, setting up a Gradle project, using ByteBuddy to instrument methods with advice, registering gauge metrics, packaging the jar, handling common class‑loader pitfalls, and configuring deployment via the OpenTelemetry operator.
This article explains what Prometheus is, why TDMQ Pulsar needs Prometheus integration, outlines the two monitoring designs, and provides a detailed, step‑by‑step guide with configuration examples to expose Pulsar metrics for Prometheus collection.
This article traces the two‑decade evolution of message queues—from early decoupling solutions like ActiveMQ, through high‑throughput designs such as Kafka, to modern platformized systems like RocketMQ and Pulsar—while explaining fundamental concepts, partitioning, and storage architectures that underpin today’s distributed messaging platforms.
This article introduces five widely used message middleware systems—Kafka, RabbitMQ, ActiveMQ, RocketMQ, and Pulsar—explaining their core architectures, key components, typical use cases, and advantages for building reliable, high‑throughput backend solutions.
This article explains what message middleware is, outlines its key use cases such as asynchronous communication and decoupling, details core principles and models like point‑to‑point and publish/subscribe, reviews popular solutions (Kafka, RabbitMQ, RocketMQ, Pulsar, etc.), and offers selection guidance.
This article explains seven typical message‑queue use cases—including ordinary, ordered, delayed, transactional, trace, dead‑letter, and priority messages—detailing their business motivations, implementation challenges, and concrete code examples for Kafka, RocketMQ, Pulsar, and RabbitMQ.
From the early days of ActiveMQ and RabbitMQ to the high‑throughput designs of Kafka, RocketMQ, and Pulsar, this article traces the development stages of message queues, explains core concepts such as topics, producers, consumers, and partitions, and compares their architectural and storage designs.
To give the business team a global, application‑level view of Pulsar performance, the team built a SkyWalking Java‑Agent plugin that automatically collects producer and consumer metrics from the Pulsar client, exposing latency, backlog and failure counts via Prometheus without modifying the client code.
This article traces the development of message queues from early decoupling solutions like ActiveMQ and RabbitMQ, through high‑throughput designs such as Kafka and RocketMQ, to modern platform‑centric systems like Pulsar, while detailing core concepts, architecture diagrams, storage mechanisms and trade‑offs.
The article traces two decades of message‑queue evolution—from early decoupling tools like ActiveMQ, through Kafka’s high‑throughput log model, to Pulsar’s layered, cloud‑native architecture—explaining core concepts, storage designs, and trade‑offs that guide choosing the right MQ for modern workloads.
This article reviews four emerging Linux code editors—Pulsar, Atom Community, Lapce, and Lite XL—highlighting their origins, features, performance, and suitability as potential alternatives to Visual Studio Code for developers seeking lightweight or community‑driven tools.
This article provides a comprehensive overview of major message‑queue middleware—including RocketMQ, Kafka, Pulsar, and RabbitMQ—covering fundamental concepts such as tags, groups, offsets, architectural components, storage mechanisms, transaction workflows, rebalance strategies, and recent developments, while comparing their features and performance characteristics.
vivo’s Internet Storage team details how it operates RocketMQ for low‑latency online services and Kafka for massive big‑data pipelines, outlines resource isolation, traffic balancing, intelligent throttling, and governance practices, and describes its migration from RabbitMQ and planned shift from Kafka to cloud‑native Pulsar.
This article introduces Apache Kafka and Apache Pulsar, compares their core features such as publish/subscribe messaging, storage, real‑time pipelines, and stream processing, outlines key characteristics like high throughput, scalability and fault tolerance, and explains fundamental concepts and architecture components unique to each platform.
This article explains how Apache BookKeeper's garbage collection works, detailing minor and major GC triggers, compression strategies, entry log size management, usage calculations, and provides a step‑by‑step code analysis of the GarbageCollectorThread implementation.
The article explains fundamental concepts, common use cases, and selection criteria for four popular message‑queue platforms—Kafka, RabbitMQ, Pulsar, and RocketMQ—detailing each system’s architecture, strengths, and drawbacks so readers can match workload, scalability, ordering, latency, and operational needs to the most suitable middleware.
This article surveys common delayed‑message implementations—including external storage, RocksDB, Redis, and open‑source MQs like RocketMQ, Pulsar, and QMQ—detailing their architectures, advantages, disadvantages, and practical considerations for distributed asynchronous messaging systems.
This article surveys common delayed‑message solutions in distributed asynchronous messaging, evaluates implementations based on external storage, databases, RocksDB, Redis, and open‑source MQs like RocketMQ, Pulsar and QMQ, and discusses their advantages, drawbacks, and practical design considerations.
This article provides a comprehensive technical overview of Apache BookKeeper, covering its role as a distributed append‑only log service, core concepts, high‑availability mechanisms, storage‑media evolution, comparisons with Raft, and community resources, while illustrating its use in Pulsar and large‑scale data platforms.
This article surveys common delayed message implementations—including external storage, RocksDB, Redis, and open‑source MQs like RocketMQ, Pulsar, and QMQ—analyzing their architectures, advantages, drawbacks, and practical considerations for building reliable distributed asynchronous messaging systems.
This article examines various delayed‑message implementations in distributed message‑queue systems, comparing external‑storage approaches using databases, RocksDB and Redis, and reviewing built‑in solutions in open‑source MQs such as RocketMQ, Pulsar and QMQ, while discussing their advantages, drawbacks and design considerations.
This guide compares Kafka, Pulsar, RocketMQ, RabbitMQ, and NSQ across architecture, consumption models, delay and dead‑letter support, persistence, ordering, multi‑tenant isolation, and performance, recommending Kafka for high‑throughput log processing, Pulsar for transactional delayed messaging and disaster recovery, and NSQ for Go‑centric custom solutions.
In a detailed interview, Tencent Cloud micro‑service director Wang Hongzhi outlines middleware’s expanding scope, cloud‑native adaptation, standardization challenges, service‑mesh innovations, and the evolving landscape of messaging systems like Kafka, Pulsar, and RocketMQ.
This article surveys common delayed‑message solutions in distributed messaging, compares implementations based on external storage (MySQL, RocksDB, Redis) and built‑in MQ features (RocketMQ, Pulsar, QMQ), and analyzes their advantages, disadvantages, and design considerations.
The article introduces RoP 0.2.0, a redesign of RocketMQ on Pulsar that eliminates message loss, duplicate consumption, and partial partition consumption, adds ACL authentication, partition expansion for higher throughput, native RocketMQ management APIs, refactors MessageID and routing models, and outlines future features such as tracing, delayed messages, and advanced monitoring.
This article compares Pulsar with Kafka, RocketMQ and InLong‑TubeMQ, focusing on cloud‑native multi‑tenant design, multi‑level flow control, compute‑storage separation, partitioning strategies, metadata storage, and message delivery mechanisms, highlighting each system's architectural advantages and trade‑offs.
Pulsar’s cloud‑native, multi‑tenant architecture with hierarchical flow control, stateless brokers separated from BookKeeper storage, global partitions supporting multiple consumers, push‑based dispatch, and precise acknowledgment outperforms Kafka, RocketMQ, and InLong‑TubeMQ, though its reliance on ZooKeeper and higher resource usage demand careful operational planning.
The article compares how RocketMQ, Apache Kafka, and Apache Pulsar implement transactional messaging—each using a two‑phase commit with half‑messages or transaction buffers, distinct coordinators, and idempotent producers—to provide atomicity and either strong consistency (RocketMQ) or exactly‑once delivery for high‑throughput stream processing (Kafka and Pulsar).
This article explains CDC fundamentals, compares query‑based and log‑based capture, describes typical CDC‑to‑lake architectures using Pulsar and Hudi, dives into Hudi's core design, optimization techniques, and future roadmap, and provides practical insights for building scalable data lakes.
This article explains the evolution of message‑queue middleware by comparing the basic double‑ended queue implementation, Redis list usage, Kafka’s partitioned log architecture with segment files and sparse indexes, and Pulsar’s compute‑storage separation using BookKeeper, highlighting their designs, strengths, and trade‑offs.
This article compares popular message middleware such as Redis, Kafka, and Pulsar, explaining their underlying data structures, strengths and weaknesses, and how concepts like partitions, replication, cursors, and storage segmentation enable high performance, scalability, and reliability in modern distributed messaging systems.
This article compares Redis list, Kafka, and Pulsar as messaging middleware, explaining their architectures, strengths, and weaknesses—including queue fundamentals, partitioning, cursor management, consumer groups, high‑availability mechanisms, storage strategies, and consumption models—to help readers choose the right solution for large‑scale systems.
This article explains the evolution of message‑queue middleware by comparing basic double‑ended queue implementations, Redis list usage, Kafka’s partitioned log design with cursor‑based consumption and high‑availability replication, and Pulsar’s compute‑storage separation using BookKeeper, while highlighting their strengths, limitations, and practical trade‑offs.
This article compares the fundamentals and limitations of Redis list‑based queues with Kafka and Pulsar, explaining how partitions, cursors, replication, segment storage, and consumption models overcome Redis’s drawbacks and enable high‑performance, scalable, and highly available messaging systems.
This article compares the fundamentals and design trade‑offs of popular message‑queue middleware—Redis list, Kafka, and Pulsar—explaining their data structures, partitioning, persistence, consumer models, high‑availability mechanisms, and scalability challenges for developers and architects.
The article compares basic in‑memory queues, Redis lists and streams, Kafka’s partitioned log architecture, and Pulsar’s compute‑storage separation, explaining each system’s core mechanisms, strengths, and limitations so readers can choose the most suitable message‑queue solution for their workloads.
This article explains how Tencent’s cloud‑native message queue TDMQ, built on Apache Pulsar’s storage‑compute separation, meets financial‑grade reliability, strong consistency, horizontal scalability, and cross‑region disaster‑recovery requirements through a quorum‑based consistency model, multi‑protocol support, and read‑only broker design.
This article explores the Pulsar C++ client’s hand‑crafted Future and Promise classes, comparing them with C++11’s std::future/std::promise, illustrating thread creation with pthread and std::thread, and detailing the internal design, callbacks, synchronization, and usage patterns for asynchronous programming.
The article explains Kafka's consumer‑group rebalance and Pulsar's unified queue/stream subscription models, compares their partition assignment strategies, and demonstrates both with Docker‑based Pulsar setups, Java consumer code, and practical failover and exclusive scenarios.
