This article explains common delayed‑queue scenarios, demonstrates how to implement a delay queue using Redis Zset with atomic Lua scripts, and shows how to deploy and use Beanstalkd—including Java client setup and sample producer/consumer code—while offering extensions and best‑practice tips.
This article provides a detailed side‑by‑side comparison of five popular message‑queue systems—Kafka, RabbitMQ, ZeroMQ, RocketMQ and ActiveMQ—covering documentation, language support, protocols, storage, transactions, load balancing, clustering, management UI, availability, duplication handling, throughput, subscription models, ordering, acknowledgments, replay, retry mechanisms and concurrency.
This article compiles essential Kafka interview material, covering its role as a message queue, usage scenarios, architectural components, storage mechanisms, consumer group rebalancing, high‑availability features, replication details, ordering guarantees, producer/consumer client design, topic management, log retention, performance optimizations, and key monitoring metrics.
This article details ByteDance's (今日头条) migration to RocketMQ, explaining the business drivers, the reasons for choosing RocketMQ, the proxy‑based architecture for producers and consumers, performance gains, operational challenges, and the disaster‑recovery designs implemented to ensure high availability in a micro‑service environment.
This article explains how to evolve a simple single‑node backend by adding a reverse proxy, introducing load balancers, scaling databases, adopting micro‑services, leveraging caches and CDNs, using message queues, and applying partitioning techniques to handle massive traffic while maintaining consistency and reliability.
TubeMQ is a trillion‑level, Java‑based distributed message‑queue middleware designed for massive‑data ingestion, offering 140 k TPS with sub‑5 ms latency, high reliability, low cost, and horizontal scalability, and is being open‑sourced to the Apache foundation to foster community collaboration and future expansion beyond traditional MQ functions.
This article details NetEase Cloud Music's journey from evaluating RabbitMQ, Kafka, and RocketMQ to designing a fully controllable, high‑availability message queue with failover, tracing, monitoring, and numerous custom extensions that now serve hundreds of services and billions of messages daily.
This article explains the role of message‑queue middleware in distributed systems, covering its communication patterns, ten key advantages such as decoupling and scalability, and practical use cases like asynchronous processing, traffic shaping, log handling, and e‑commerce architectures.
This article provides a comprehensive overview of RocketMQ, covering its architecture, how it handles ordered and duplicate messages, transaction processing, producer and consumer mechanisms, storage design, subscription models, and practical best‑practice recommendations for building reliable distributed messaging systems.
This article explains how Zookeeper functions as an essential part of Kafka by managing broker status, controller election, quotas, ISR tracking, node and topic registration, as well as consumer offset storage and registration, providing a comprehensive overview for interview preparation.
This article details Xiaomi's journey from Kafka 0.8 to the home‑grown Talos system, covering business motivations, storage‑compute separation architecture, key challenges such as tail‑read and consistency, and extensive performance, resource, and platform optimizations that enable a high‑throughput, multi‑tenant messaging service.
This article provides a comprehensive overview of Apache Kafka, covering its fundamental capabilities, typical use cases, core components, key APIs, and essential concepts such as topics, partitions, segments, brokers, producers, and consumers, illustrated with diagrams.
This article explains the fundamental properties of message queues, why they are needed, the advantages they bring such as improved response speed, stability, and decoupling, and discusses distributed system challenges and practical implementation considerations for selecting and deploying a suitable message‑queue solution.
This article provides a comprehensive guide to using Spring‑Kafka, covering simple integration, embedded Kafka for testing, creating topics programmatically, advanced KafkaTemplate usage, transactional messaging, ReplyingKafkaTemplate for request‑reply, listener configurations, manual acknowledgments, lifecycle control, message forwarding, and retry with dead‑letter queues.
This article provides a comprehensive 17‑point comparison of five popular message queue systems—Kafka, RabbitMQ, ZeroMQ, RocketMQ, and ActiveMQ—covering documentation, supported languages, protocols, storage, transactions, load balancing, clustering, management UI, availability, message duplication, throughput, subscription models, ordering, acknowledgments, replay, retries, concurrency, and more.
This article details ByteDance’s adoption of RocketMQ within its massive microservice ecosystem, explaining the business challenges, reasons for choosing RocketMQ, the proxy‑based deployment architecture, encountered technical issues, and the multi‑datacenter disaster‑recovery strategies implemented to ensure high availability and performance.
This article explains the Redis concurrent key competition problem, illustrates typical scenarios where multiple clients modify the same key, and presents four practical solutions—optimistic locking with WATCH/EXEC, distributed locks, timestamp ordering, and message‑queue serialization—to ensure data correctness under high concurrency.
This article examines five common Java backend pitfalls that startups face—single‑machine locking, improper multithreading, poorly defined workflows, unsafe system interactions, and non‑paginated queries—explains why they cause failures in distributed environments, and provides practical solutions such as distributed locks, message queues, state‑machine redesign, RPC interfaces, and pagination techniques.
This article summarizes Shen Hui’s presentation on how ByteDance adopted RocketMQ in a massive micro‑service environment, detailing the business background, reasons for choosing RocketMQ, the proxy‑based deployment, encountered challenges, and the multi‑data‑center disaster‑recovery solutions implemented.
This article explains how to design a high‑concurrency flash‑sale (秒杀) system by leveraging browser and CDN caching, read‑write split Redis for traffic filtering, Lua scripts for atomic stock deduction, master‑slave Redis for fast inventory updates, and Redis‑based message queues for asynchronous order processing.
This article explains how using message queues for asynchronous communication decouples services, improves fault tolerance, and enhances performance, illustrated with a real‑world early‑education app case and a guide to employing dead‑letter queues for handling failed messages.
The article analyzes a common interview question about guaranteeing message ordering in message‑queue systems, explains why order matters using MySQL binlog and MQ examples, illustrates scenarios where ordering breaks in RabbitMQ and Kafka, and proposes concrete architectural solutions.
This article walks through constructing a lightweight instant‑messaging backend, covering version 1.0.0 features, reliability guarantees, application‑level ACK handling, security encryption, database schema for users, relations and offline messages, and storage strategies to prevent duplicate delivery.
This article walks through constructing a lightweight, feature‑rich instant‑messaging backend (IM1.0.0) that supports one‑to‑one text/file messaging, delivery/read receipts, LDAP login, horizontal scaling via connector and transfer modules, user‑status management with Redis, offline storage using MySQL and message queues, and outlines the overall architecture.
The article explains how WMB, a high‑performance distributed message queue, uses Paxos‑based master election, periodic lease renewal, and a multi‑stage load‑balancing strategy to evenly distribute master nodes across groups, improve throughput, and ensure consistent message delivery.
This article details a real‑world RocketMQ case where expanding a topic's queue count caused two consumer groups to miss messages on one broker, explains the root cause of missing subscription metadata after cluster scaling, and outlines the manual steps taken to restore full consumption.
Kafka is a distributed, partitioned, replicated messaging system originally developed by LinkedIn, offering high throughput, low latency, fault tolerance, and scalability; this article explains its core concepts, file storage design, partition replication, leader election, consumer groups, delivery guarantees, and operational considerations for big‑data pipelines.
This article introduces QMQ, Qunar's internal distributed message queue, covering its background, design motivations, core concepts, code examples for producing and consuming both real‑time and delayed messages, transactional messaging support, and the overall architecture of its metaserver, broker, and delay components.
This article provides a comprehensive overview of Kafka, covering its basic concepts, architecture, deployment, configuration, monitoring, producer and consumer settings, offset management, high availability, replication, leader election, and practical tips for deployment, tuning, and troubleshooting in production environments.
This guide explains how to set up ThinkPHP5's built‑in Queue component with a Redis driver, create a custom queue class, configure the queue, produce messages, and run a consumer process, including handling retries, delays, and daemon mode.
This article compares Apache Kafka and Apache Pulsar, detailing Kafka's scalability challenges, Pulsar's architectural benefits, performance gains, multi‑tenant support, security features, and provides code examples and migration guidance for large‑scale streaming applications.
This article explains Kafka's design background, persistence mechanisms, disk sequential I/O optimizations, network and compression strategies, and stability features such as partitioning, replication, and ISR, illustrating how these techniques enable high‑throughput, low‑latency real‑time log processing in big‑data environments.
The article explains how delayed message queues replace inefficient scheduled‑task scans in distributed systems, outlines common use cases such as order timeouts and retries, compares RabbitMQ, RocketMQ, Kafka, ActiveMQ and Redis implementations, and details NetEase’s ActiveMQ‑based solution with idempotent processing and traceability.
This article explains a practical approach to performance‑testing Dubbo add and delete methods of a message queue by pre‑generating payloads, using a thread‑safe LinkedBlockingQueue, and driving the calls with JMeter‑compatible Java code.
This article explores Redis beyond its traditional caching role, covering distributed locks, message queues, bitmaps, Bloom filters, geolocation, and rate limiting, while highlighting typical implementation patterns, code examples, and the inherent limitations of each scenario.
This article describes how to conduct load testing of Dubbo queue service methods for adding and deleting messages using Java concurrency utilities and JMeter, including code examples for constructing test data, managing a thread‑safe LinkedBlockingQueue, and handling initialization challenges during repeated test runs.
This article explains why maintaining message order is critical, describes common scenarios that cause ordering problems in RabbitMQ and Kafka, and presents practical strategies such as queue partitioning, single‑consumer designs, and internal memory queues to guarantee ordered consumption.
This article explains the core principles of message queues, outlines common data‑loss scenarios for RabbitMQ and Kafka, and provides practical techniques—such as transactions, confirm mode, persistence settings, and replication configurations—to ensure reliable, loss‑free messaging.
This guide defines and explains essential AMQP terminology—including virtual hosts, queues, exchanges, bindings, routing keys, producers, consumers, connections, and channels—to help developers grasp how message routing and queue management work in modern messaging systems.
This guide explains why work queues are needed, compares round‑robin and fair dispatch strategies in RabbitMQ, and provides complete Java examples for producers and multiple consumers using basicQos and manual acknowledgments to achieve balanced task distribution.
The article explains why message queues are employed for decoupling, asynchronous processing, and load‑shedding, outlines their advantages and disadvantages, and compares popular MQ products such as Kafka, ActiveMQ, RabbitMQ, and RocketMQ to guide technology selection.
This guide explains the basic RabbitMQ model, shows how to obtain a Java connection, demonstrates code for a producer to send messages and a consumer to receive them, and discusses the limitations of a simple queue setup.
This article explains the design and implementation of NSQ dual‑ and multi‑data‑center architectures using a lookup‑migrate proxy, covering deployment scenarios, routing strategies, migration phases, JSON response transformations, and practical lessons learned for reliable message publishing and consumption across data centers.
This guide explains how RabbitMQ exchanges—Direct, Topic, Headers, and Fanout—route messages, compares their performance, and provides practical examples and configuration tips for efficient backend messaging.
This article explains why Apache Kafka has become a cornerstone of modern big‑data pipelines by detailing its high‑throughput, fault‑tolerant publish‑subscribe architecture, real‑time processing capabilities, extensive language support, scalability mechanisms, and the wide range of use cases adopted by leading enterprises.
This guide walks you through installing RabbitMQ via Docker, configuring ports and users, exploring its messaging model, and integrating it with PHP using the php‑amqplib client, complete with runnable code examples for publishing and consuming messages.
This article provides a comprehensive overview of Apache RocketMQ’s core architecture, including topic routing mechanisms, message storage file designs, high‑availability message sending, concurrent pull and consumption processes, HA synchronization, and transaction messaging, while offering practical learning steps and programming techniques for developers.
This article explains the concept of high availability for message queues, analyzes interview expectations, and details the HA mechanisms of RabbitMQ (including single, normal cluster, and mirrored modes) and Kafka (partition replication and leader election), highlighting their advantages, drawbacks, and practical considerations.
A new employee at an OTA company faced a mysterious outage where thousands of payment‑related messages piled up in the queue, the system auto‑recovered, and a detailed investigation revealed a stuck MySQL transaction caused by missing response timeout settings, leading to lock contention and message backlog.
This article introduces RocketMQ, compares it with ActiveMQ, explains its producer/consumer group model, describes various broker deployment modes, and provides a step‑by‑step dual‑master installation guide with configuration, log, and console setup instructions.
This article provides a comprehensive overview of Apache Kafka's architecture, covering topics such as producer and consumer workflows, partition and replica management, leader election, offset handling, message delivery semantics, transaction support, and file organization, illustrating how Kafka achieves high performance and scalability.
This article explores Apache Kafka’s evolution, core messaging and stream processing capabilities, typical use cases, internal storage mechanisms, API choices, and best practices for deploying Kafka on Kubernetes, providing readers with comprehensive guidance to assess suitability and implement effective Kafka solutions.
This article explains what RabbitMQ is, why it is used, typical application scenarios, how to ensure reliable message publishing and consumption, the roles of sender and receiver acknowledgments, routing mechanisms, clustering options, advantages, drawbacks, and provides interview‑style Q&A with a keyword to obtain full answers.
This article introduces Apache Kafka as a distributed streaming platform, explaining its three core capabilities, key concepts such as producers, topics, brokers, partitions and consumers, and detailing how messages are delivered, stored in partitions, and consumed by consumer groups.
This article analyzes the challenges of scaling group chat in an instant messaging system and presents architectural optimizations—including shared message storage, periodic conversation list updates, offline count handling, and version‑based incremental sync—to reduce write and read amplification while improving overall performance.
This article explains what a message queue (MQ) is, why it is needed beyond in‑memory Java queues, and how it enables decoupling, asynchronous processing, peak‑shaving and rate‑limiting, while also discussing high‑availability, data‑loss, and consumer‑side considerations.
This article explores the fundamental design of a message queue system, covering overall architecture with producers, brokers, and consumers, key design considerations for the broker, communication protocols such as JMS and AMQP, storage options, and strategies for managing consumption relationships and advanced features.
This article provides a step‑by‑step guide on installing Apache Kafka, creating topics, producing and consuming messages via command line, and demonstrates how to integrate Kafka with Apache Flink using the Flink‑Kafka connector, custom serialization schemas, and event‑time window processing.
This guide explains how to construct a client library for Youzan's customized NSQ, covering workflow configuration, nsqd discovery via nsqlookupd, connection establishment, message publishing and consumption, ordered consumption, and the implementation of message tracing with concrete command syntax and response handling.
This article provides a comprehensive overview of NSQ, covering its core components, message flow, producer and consumer interactions, key configuration parameters, handling of timeouts, backoff mechanisms, service discovery with nsqlookupd, and practical considerations for scaling and reliability.
This article introduces RocketMQ, explains its architecture and core components, and guides readers through creating and sending test messages using Java code and a Python‑JPype approach, ultimately proposing a simple web‑based tool for cross‑language message simulation.
The article explains how MaFengWo built a custom travel transaction system that uses multi‑order distribution, a grab‑pool message queue, concurrency‑controlled locking, and long‑connection notifications to efficiently match personalized travel requests with multiple suppliers, boosting conversion rates and platform performance.
This article presents a comprehensive case study of how JD's Service Market improved the performance of its core "Available Plugin List" service by analyzing self‑optimizations, resolving external resource conflicts, designing a unified caching model, and implementing asynchronous cache construction using binlog and MQ, ultimately achieving a six‑fold traffic surge handling during peak events.
While message queues can decouple services and smooth traffic spikes, they also introduce availability risks, stability problems like message loss or duplication, and distributed consistency challenges that require careful high‑availability design and reliable transaction strategies.
This article breaks down common interview questions about message queue middleware, compares popular options like RabbitMQ, RocketMQ, and Kafka, and explains why introducing a message broker improves system decoupling, enables asynchronous processing, and helps handle traffic spikes efficiently.
Didi Mobility has open‑sourced DDMQ, an enterprise‑grade distributed message queue that delivers millisecond‑level latency and millions of messages per second, supports real‑time, delayed and transactional messages via RocketMQ/Kafka and RocksDB, offers multi‑language SDKs, flexible consumption, server‑side filtering, a unified console, and achieves five‑nine availability across Didi’s ride‑hailing, finance and delivery platforms.
This article explains how RocketMQ’s transactional message feature enables distributed transaction final consistency by using half messages, message check‑back, and defined transaction states, and provides complete Java code samples for producers, listeners, and consumer idempotency handling.
This article provides a comprehensive overview of RocketMQ, covering its origins and evolution, core architecture components, message domain model, essential features such as ordered delivery and deduplication, and typical use cases like traffic shaping, asynchronous decoupling, and distributed transaction messaging.
The article presents an in‑depth technical overview of JD's self‑developed JMQ middleware, detailing its performance improvements over previous versions, comparing write throughput with Kafka under synchronous and asynchronous scenarios, and explaining JMQ4's storage architecture, dense indexing, high‑performance I/O, and cache management strategies.
This article explains how to design and implement a reliable message service that ensures atomicity between upstream business transactions and downstream message delivery, using status management, callbacks, and scheduled tasks to achieve eventual consistency in distributed systems.
This article explains the core concepts, components, design considerations, and common models of message queues—including point‑to‑point and publish/subscribe—while covering ordering, acknowledgment, transaction support, persistence, high availability, and practical selection guidance for high‑concurrency systems.
This tutorial demonstrates how to create a custom error‑handling flow for Spring Cloud Stream by building a simple producer‑consumer example, configuring input/output bindings, using @StreamListener to simulate a failure, and routing the exception to a @ServiceActivator fallback method, while also discussing version‑specific limitations.
This article examines the role of message‑queue middleware in high‑concurrency IM systems, compares popular open‑source options such as ActiveMQ, RabbitMQ, Kafka, RocketMQ and ZeroMQ, and provides a detailed multi‑dimensional framework—including functionality, performance, reliability, operational management, and ecosystem factors—to help engineers select the most suitable queue for their specific business needs.
This article provides a comprehensive technical overview of Kafka, covering its core concepts, producer and consumer models, architecture, configuration parameters, replication mechanisms, performance optimizations, operational monitoring, tooling scripts, and related product implementations for real-time data processing.
The article surveys thirty‑nine essential open‑source tools for constructing microservices—from API management, messaging, and monitoring to Kubernetes orchestration, programming languages, frameworks, and serverless platforms—highlighting how each solution supports communication, scalability, and rapid development across enterprise, government, education, and charitable applications.
This article provides a comprehensive guide to RabbitMQ, covering the fundamentals of message middleware, the P2P and Pub/Sub models, a comparison with Kafka and RocketMQ, detailed explanations of exchanges, queues, channels, and advanced features such as mandatory routing, backup exchanges, TTL, dead‑letter queues, delayed queues, priority queues, and RPC implementations for reliable distributed systems.
Redis is an open-source, in‑memory key‑value store written in C that supports multiple data structures, offers high‑performance caching, persistence via RDB and AOF, various clustering solutions such as Codis and native Redis Cluster, and is widely used for caching, ranking, session management, queues, and pub/sub.
This article explains the ACID properties of transactions, contrasts them with the CAP theorem and BASE model, and then explores practical distributed‑transaction solutions such as two‑phase commit, TCC, and reliable asynchronous messaging, including their advantages, drawbacks, and implementation tips.
At Didi, the team replaced a chaotic mix of Kafka, Redis, and other queues with a custom, RocketMQ‑based service, using dual‑write and dual‑read migration, extensive performance testing, custom failover, batch extensions, and operational tweaks to achieve stable high‑throughput, low‑latency messaging at massive scale.
This article explains how to build a high‑concurrency flash‑sale (秒杀) system by separating static page traffic with CDN, using read‑write‑split Redis for early request filtering, applying Lua scripts for atomic inventory deduction, and employing Redis as a simple message queue for asynchronous order persistence.
This article recounts Didi's journey from a chaotic mix of Kafka, RocketMQ, Redis, and other queues to a unified, high‑performance messaging platform built on Apache RocketMQ, covering the reasons for abandoning Kafka, the architecture evolution, migration strategies, performance benchmarks, and operational enhancements.
The article analyzes the massive technical challenges of Alibaba's Double 11 flash‑sale event and explains how cloud elasticity, distributed messaging, containerization, real‑time data processing, AI, front‑end optimization, caching, monitoring, and database sharding together enable billions of transactions within minutes.
This article introduces Apache RocketMQ by covering its origin, core concepts such as topics, producers and consumers, storage architecture with CommitLog and ConsumeQueue, deployment components like brokers and name servers, and practical best‑practice guidance for handling duplicates, ordering, and message replay.
This article examines the core features, performance characteristics, and typical application scenarios of popular message queue middleware such as ZeroMQ, RabbitMQ, ActiveMQ, Redis, Kafka, and others, while also discussing when to adopt a message queue, its benefits like decoupling, eventual consistency, broadcasting, and flow control, and summarizing best‑practice guidelines.
This article explains the fundamentals of distributed transactions, why they arise in modern micro‑service architectures, outlines the ACID properties, presents typical use cases such as payments and order processing, and compares common implementation approaches like 2PC, message‑based eventual consistency, and TCC.
This article examines the high‑real‑time requirements of incremental data ingestion for search engines, compares three update schemes, and details how adopting a Kafka subscription‑based message‑queue approach dramatically improves latency and flexibility for the Nuomi search framework.
This article reviews the evolution of Alibaba's Aliware message engine, analyzes the latency and availability challenges faced during Double‑11, and describes the low‑latency optimizations, capacity‑guarantee strategies, and multi‑replica high‑availability architecture implemented in RocketMQ to sustain trillion‑level message flows.
This article outlines the progressive evolution of system architecture—from a single‑server LAMP setup through service and data separation, caching, clustering, read/write splitting, CDN, distributed storage, NoSQL, business decomposition, and finally distributed services with messaging, service frameworks, and governance—highlighting the motivations, characteristics, and challenges at each stage.
This guide walks through Redis 5's stream feature, showing how to set up a Docker Redis instance, add entries, perform range queries, use blocking reads, create and manage consumer groups, handle pending messages, claim ownership, inspect stream metadata, delete entries, and limit stream length for efficient message queue implementations.
NSQ is a lightweight, high‑performance open‑source message queue written in Go, and this article explains its core components (nsqd, nsqlookupd, nsqadmin, utilities), message flow architecture, configuration details, deployment recommendations, and additional considerations such as persistence, delivery guarantees, and scalability.
This article evaluates the server‑side performance of three popular message middleware platforms—Kafka, RabbitMQ, and RocketMQ—by measuring throughput and latency when sending small 124‑byte messages, revealing that Kafka leads, followed by RocketMQ and then RabbitMQ.
The article reviews the ten‑year evolution of Ant Financial's message queue, detailing its core reliability, consistency, availability and performance requirements, the architectural mechanisms built to meet them, the shift to pull‑mode and API‑mode designs, and the recent integration of compute capabilities to create a smart data transmission platform.
This article introduces message queue middleware, explains its role in distributed systems for asynchronous processing, system decoupling, traffic shaping, log handling and message communication, and provides concrete e‑commerce and log‑collection examples illustrating how queues improve performance, scalability and reliability.
This article reviews the essential concepts of message‑queue middleware, covering why it is used, its disadvantages, how to choose among popular MQs, and practical techniques for ensuring high availability, avoiding duplicate consumption, reliable transmission, and ordered processing.
This article explains Kafka's role as a distributed message queue, covering its architecture, replication mechanisms, producer‑consumer workflow, message integrity guarantees, schema management, and performance tuning for high‑throughput, low‑latency backend systems.
To support Meituan’s rapid growth, this article examines the design and evolution of several industry-leading message-queue solutions—including Alibaba’s Notify and RocketMQ, Tencent’s Tube and Hippo, and Ctrip’s Herms—highlighting their reliability, scalability, and decoupling features, and extracting key insights for building robust MQ systems.
This article reviews essential concepts of message queue middleware, covering why to use queues, their drawbacks, selection criteria, high‑availability strategies, deduplication, reliable transmission, ordering guarantees, and practical advice for interview preparation.
This article compares the architectures of Kafka and RocketMQ, examines their availability and reliability mechanisms, evaluates their strengths and weaknesses, and proposes hybrid designs and simplified MQ architectures for building highly available and reliable messaging systems.
This article reviews essential concepts of message queue middleware, covering why to use it, its drawbacks, selection criteria, high‑availability designs, preventing duplicate consumption, ensuring reliable transmission, and maintaining message order, providing a concise study guide for developers and architects.
This article explains how message queues enable system decoupling, smooth traffic spikes, achieve eventual consistency, support broadcast consumption, and handle retry pitfalls, providing practical code examples and architectural guidelines for reliable backend design.
