This guide walks you through installing Zookeeper and Kafka on a CentOS server, configuring essential settings, creating topics, and running producers and consumers, while highlighting common pitfalls and providing the exact commands needed for a successful deployment.
This article outlines the progressive stages of system architecture—from a single‑server LAMP setup through application‑data separation, caching, clustering, read/write splitting, CDN, distributed storage, NoSQL, business decomposition, and finally distributed services—while detailing essential technologies such as message queues, service frameworks, service buses, communication patterns, and governance mechanisms like Dubbo and OSB.
This article explains how a high‑traffic e‑commerce platform uses RabbitMQ for reliable, high‑throughput order processing, covering its core concepts, routing patterns, clustering, high‑availability queues, client support, management tools, and step‑by‑step cluster deployment.
This article examines the architectures of Kafka and RocketMQ, analyzes their availability and reliability mechanisms, evaluates their strengths and weaknesses, and proposes a hybrid MQ design that combines the benefits of both systems while simplifying dependencies and improving fault tolerance.
This guide walks through installing RocketMQ on a CentOS 6.5 VM, launching NameServer and Broker, provides sample Java consumer and producer code, and outlines best‑practice tips for idempotent consumption, batch processing, message filtering, performance tuning, and reliable message delivery.
RocketMQ is a distributed queue-based middleware offering strict message ordering, diverse pull modes, scalable subscribers, real-time subscription, and massive message accumulation, with detailed network deployment, storage mechanisms, key features, flushing strategies, query methods, server-side filtering, JVM memory utilization, and solutions for message backlog.
This article details a 50‑hour experiment that simulates network packet loss between geographically separated RabbitMQ clusters and clients, showing how packet drops affect ACKs, socket buffers, and latency, and ultimately demonstrating that no messages were lost during the test.
This article tells the story of a developer who first used a blocking email call, then switched to multithreading, and finally adopted a message queue to achieve asynchronous processing, decoupling, scalability, and reliability for user registration emails.
This article explains the design and improvements of WeChat's MQ 2.0 asynchronous queue, covering its cross‑machine consumption model, enhanced task scheduling, MapReduce‑style processing, stream tasks, and robust overload protection that together enable high‑performance, resilient backend services.
The article introduces WeChat's self‑developed MQ 2.0 asynchronous queue, detailing its architecture, cross‑machine consumption model, improved task scheduling, efficient processing frameworks—including a MapReduce‑style engine and streaming tasks—and robust overload protection mechanisms that together boost reliability and performance for large‑scale backend services.
This article explains the producer‑consumer and publish‑subscribe messaging patterns, introduces RabbitMQ’s core concepts such as exchanges, queues, routing keys, and reliability mechanisms, and demonstrates how to apply these patterns to data ingestion and event distribution scenarios with practical diagrams.
This article explains Kafka's core concepts and compares producer‑consumer and publish‑subscribe models, illustrating how to apply each pattern for data ingestion and event distribution in distributed backend systems, and offers practical design alternatives when Kafka’s native capabilities fall short.
This article analyzes a frequent Android crash caused by posting delayed Runnables to a Handler, explains why it is hard to reproduce, and presents a solution that extends the delay to reliably trigger and then mitigate the issue using message‑queue manipulation.
This article outlines the design and implementation of high‑concurrency systems, covering server architecture, load balancing, database clustering, caching strategies, message‑queue based asynchronous processing, static data handling, and operational best practices such as monitoring, redundancy, and automation.
Message queues act as a crucial middleware component in distributed systems, addressing coupling, asynchronous processing, traffic shaping, and high availability, with real-world scenarios such as asynchronous handling, decoupling, traffic throttling, logging, and communication, while reviewing popular solutions like ActiveMQ, RabbitMQ, ZeroMQ, Kafka, and JMS.
This article shares practical patterns and built‑in mechanisms for dealing with duplicate messages, message ordering, concurrent updates, asynchronous acknowledgments, and batch processing in a large‑scale, message‑driven architecture, illustrated with QMQ examples from Qunar's platform.
This article explains Kafka’s overall cluster structure, how producers write messages to partitions, and how consumer groups read messages, covering topics, partitions, leaders, followers, offsets, and load‑balancing mechanisms in a concise technical overview.
This article introduces QBus, a Kafka‑derived, highly‑available message queue built for 360, detailing its origins, core features, architectural design, performance advantages over traditional queues, and Java SDK usage examples for producers and consumers.
This guide walks through the fundamentals, installation steps, configuration details, and Python integration for three essential backend services—Memcached for high‑performance caching, Redis for persistent key‑value storage, and RabbitMQ for reliable messaging—complete with command‑line examples, code snippets, and best‑practice tips.
This article examines the fundamental challenges of achieving consistency in distributed systems, explains the CAP theorem, compares two‑phase and three‑phase commit protocols, explores XA transactions, and presents practical compensation patterns such as local message tables, non‑transactional and transactional MQ designs, highlighting their trade‑offs and applicability.
Disque is an experimental, distributed, fault‑tolerant in‑memory message queue built in C that extends Redis concepts with synchronous replication, configurable delivery semantics, explicit acknowledgments, fast‑ack support, dead‑letter handling, and optional disk persistence for robust backend messaging workloads.
This article outlines the challenges of high‑traffic scenarios such as flash sales and timed red‑packet distribution, and presents a comprehensive backend architecture—including load balancing, database clustering, NoSQL caching, CDN, concurrency testing, message queues, first‑level caching, and static data strategies—to ensure smooth operation and optimal user experience.
The article explains that massive user traffic forces servers to adopt distributed systems to achieve high throughput, concurrency, low latency, and load balancing, and it details architectural patterns, concurrency models, caching, storage, coordination, messaging, transaction, deployment, and monitoring techniques for building and managing such systems.
The article explains the drawbacks of traditional two‑phase commit distributed transactions—reduced availability, performance, and scalability—and proposes a message‑queue‑plus‑state‑table pattern with idempotent processing to achieve eventual consistency while improving system throughput.
The article explains how insecure serialization in message‑queue middleware, especially Python's pickle used by Celery, can be abused to inject malicious payloads that trigger remote code execution on distributed workers, and it demonstrates detection and exploitation techniques against vulnerable Redis and MongoDB brokers.
This guide explains how to design and test high‑concurrency systems by choosing appropriate server architectures, load‑balancing, database clustering, caching layers, message queues, first‑level caches, and static data strategies to ensure smooth operation under heavy user traffic.
This article explains the true purpose of message queues, classifies them into broker‑based and broker‑less families, compares Kafka, RabbitMQ, and ZeroMQ in terms of performance, flexibility, and lightweight distribution, and clarifies that MQs can support both asynchronous and synchronous communication.
Discover how 360’s private cloud platform leverages SaltStack’s Syndic architecture and a Redis Pub/Sub bridge to efficiently manage and execute commands across tens of thousands of servers, addressing multi‑datacenter challenges, reducing message loss, and improving automation performance.
This article details the three-stage evolution of Xiaomi's e‑commerce architecture, introduces the design of the Notify asynchronous message system built on Redis and MySQL, and explains the subsequent upgrades—including agent proxy, Go‑based modules, and MyCAT integration—to improve scalability, reliability, and performance.
The article examines common misconceptions about game server performance, outlines essential message pipelines, introduces Gate as a reverse‑proxy for low‑latency scene synchronization, discusses integrating a message‑queue (MQ) for decoupled services, and presents a service‑oriented RPC framework with patterns to build maintainable, scalable multiplayer back‑ends.
This article outlines the concepts, classifications, and detailed backend design of a message/notification system, including architecture, database schemas, subscription mechanisms, push/pull strategies, and service-layer APIs for creating, pulling, and managing announcements, reminders, and messages.
The article explains why and when to use message queues, then walks through designing one from scratch, covering decoupling, eventual consistency, broadcast, flow control, RPC protocols, high availability, storage choices, consumer relationships, reliable delivery, transactions, performance optimizations, and push versus pull models.
The article outlines how to design a custom message queue by first identifying appropriate use‑cases such as decoupling, eventual consistency, broadcasting and peak‑shaving, then examining push versus pull models, high‑availability, ordering, duplicate handling, storage choices, batch processing, flow‑control and performance optimizations, with advanced topics reserved for a follow‑up.
This article explains the Pull mode of Pushlets, detailing its protocol steps, server-side session and queue management, client interactions, key configuration parameters, and the encapsulation of Pushlets into a Channel service for scalable backend message pushing.
This article explores microservice architecture fundamentals, compares synchronous and asynchronous communication, details RabbitMQ’s AMQP model, optimization techniques, high‑availability configurations, flow‑control mechanisms, and shares practical case studies from NetEase’s Hive platform, offering actionable insights for reliable, scalable message‑queue deployments.
This article dissects the design of 58 Daojia’s universal real‑time messaging platform, detailing the challenges of end‑to‑cloud, cloud‑to‑end, and end‑to‑end communication, the shortcomings of traditional HTTP and push solutions, and the optimized TCP, message‑bus, RPC, and protocol extensions that achieve scalability, reliability, and cross‑account chat.
The article examines various messaging patterns—Message Channel, Publisher‑Subscriber, Message Router, and others—explaining their principles, advantages, and implementation details, and illustrates their use in enterprise distributed architectures through concrete code examples and case studies such as CIMS and a medical system.
Will Larson’s insights on building scalable systems are distilled here, covering horizontal scalability, redundancy, load balancing strategies, various caching techniques, CDN usage, offline processing with message queues, Map‑Reduce for big data, and the benefits of introducing a dedicated platform layer for robust backend architecture.
This article explains the design principles, architecture, and performance optimizations of Alibaba's Notify and MetaQ message middleware, illustrating how they achieve reliable asynchronous communication, high scalability, and low latency for billions of messages during peak e‑commerce events like Double 11.
The article breaks down Twitter’s core business of following and being followed, explains its three‑tier presentation‑logic‑data architecture, details cache strategies, message‑queue isolation, and the push‑pull trade‑offs, illustrating how these components work together to achieve high‑performance, fault‑tolerant service at massive scale.
This article explains the JMS standard, its two messaging models (Point‑to‑Point and Publish/Subscribe), how messages are consumed synchronously or asynchronously, the core JMS programming objects, and provides an overview of common middleware such as ActiveMQ, RabbitMQ, ZeroMQ, and Kafka.
This article explains the design, architecture, and implementation details of a Java-based delayed queue, covering use cases, core components, message lifecycle, protocol, current topology, shortcomings, and future improvements for reliable backend processing.
This article examines Ctrip's Hermes asynchronous messaging system, detailing its evolution from a simple Mongo‑backed queue to a broker‑centric, partitioned architecture with lease‑based cluster management, and shares practical techniques for building high‑performance, low‑latency message queues in large‑scale distributed environments.
This article presents a detailed overview of Ctrip's Hermes message queue system, covering its architectural evolution from a simple Mongo‑based design to a broker‑centric, multi‑storage solution with meta‑server coordination, and discusses practical techniques for building high‑performance, scalable messaging infrastructure.
This article introduces the fundamentals of message queues, explains common use cases such as asynchronous processing, system decoupling, traffic shaping, and log handling, and reviews popular middleware implementations including JMS, ActiveMQ, RabbitMQ, ZeroMQ, and Kafka.
Kafka persists messages on disk by organizing each topic into multiple partitions, which are further divided into segment files containing paired .index and .log files; this structure enables efficient storage, offset-based lookup, and fast retrieval of specific messages through binary search across segment indexes.
This article provides a comprehensive overview of message queues, covering their core concepts, key application scenarios such as asynchronous processing, system decoupling, traffic shaping, log handling, and communication, and examines popular middleware like ActiveMQ, RabbitMQ, ZeroMQ, and Kafka, along with JMS models and programming details.
This article explains the challenges of ensuring data consistency across multiple databases during operations like Alipay transfers, introduces local and distributed transactions, details the two‑phase commit protocol, and shows how reliable messaging can replace heavyweight distributed transactions for high‑concurrency systems.
The article explains how to maintain data consistency when updating related tables by using local transactions for single‑node databases, distributed two‑phase commit for multi‑node systems, and reliable message‑queue patterns—including deduplication techniques—to avoid the performance pitfalls of traditional distributed transactions.
Learn how to handle high‑concurrency log collection by buffering logs in a Redis queue, then periodically batch‑inserting them into a database, with code examples for enqueuing, processing, error rollback, and scheduling the task.
Redis lists enable simple message queues, but using LPOP/RPOP can lose messages if a consumer crashes; employing the atomic RPOPLPUSH (or its blocking BRPOPLPUSH) command safeguards delivery, and choosing blocking over non‑blocking operations reduces wasted polling and improves resource efficiency.
This article explains the fundamentals of distributed computing, covering sharding algorithms, message‑queue based task distribution, an overview of Hadoop and its MapReduce model, and the characteristics of offline batch processing for large‑scale data workloads.
This article details a backend redesign of an instant‑messaging platform, covering Java interfaces for unread messages, Redis‑based unread counters, persistent queue processing, MySQL sharding and partitioning, SQL retrieval logic, and performance tuning to achieve high insert throughput.
This article examines the rise of cloud‑native architectures, explains core concepts of Message Oriented Middleware, compares JMS and AMQP protocols, evaluates major commercial and open‑source queue products across functionality, operations management, clustering, high‑availability, stability and extensions, and presents detailed performance test results for point‑to‑point and publish/subscribe modes.
This article explains how JD.com’s three‑layer cloud architecture—storage, middleware, and elastic compute—addresses challenges of massive unstructured data, ever‑growing caches, high‑volume messaging, and rapid service deployment using home‑grown systems like JFS, Jimdb, JMQ, and JSF.
This article explains how EQueue handles broker restarts by scanning and initializing chunk files, introduces memory‑based caching strategies to avoid frequent file reads, and outlines message deletion, querying, consumer offset storage, and queue management techniques for high‑performance backend systems.
The article details the author's multi‑month effort to replace SQL Server persistence in the open‑source EQueue message queue with a high‑performance local file storage solution, covering the motivations, design considerations, chunked binary file format, write‑and‑read mechanisms, and flushing strategies.
This article chronicles the evolution of Sina's news comment platform from a single‑server C++/MySQL prototype to a multi‑node, asynchronous, Python‑based architecture, highlighting the caching, sharding, and queueing techniques that enabled the system to survive massive traffic spikes without downtime.
This article explains Apache Kafka's distributed publish‑subscribe design, core components, storage model, ZooKeeper coordination, performance benchmarks against ActiveMQ and RabbitMQ, and provides Java producer and consumer code examples for building high‑throughput messaging applications.
This article provides an in‑depth examination of Apache Kafka’s origins, design goals, core concepts such as brokers, topics, partitions, producers and consumers, compares it with other message queues, explains its storage format, configuration options, delivery guarantees, and includes practical Java code examples for partitioning and consumption.
To prevent data inconsistencies when transferring funds—like moving money from Alipay to Yu'ebao—the article explores local transactions, the limitations of two-phase commit in distributed systems, and how reliable message queues can achieve eventual consistency without sacrificing performance.
This article distills Will Larson’s insights on designing scalable systems, covering linear capacity growth, redundancy, load‑balancing strategies, multi‑layer caching, cache invalidation, offline processing with message queues, Map‑Reduce integration, and the benefits of a dedicated platform layer.
This article explores how to maintain data consistency across services by using local transactions, two‑phase commit distributed transactions, and reliable message‑queue patterns, illustrating each method with an Alipay‑to‑YuEBao transfer example and discussing their trade‑offs.
This article provides a comprehensive overview of Apache Kafka, covering its origin, core design goals, key terminology, architectural components, message routing, consumer groups, delivery guarantees, and a detailed comparison with other popular message queue systems.
The Poppen.de social site demonstrates a highly scalable backend architecture that combines Nginx, PHP‑FPM, MySQL, Memcached, RabbitMQ, CouchDB, Graphite, Red5 and Tsung to handle millions of users, high concurrency, and extensive media traffic while maintaining performance and reliability.
This article provides a comprehensive overview of Kafka, covering its background, design goals, architecture, key terminology, message routing, consumer groups, delivery guarantees, and a comparison with other popular message queue systems such as RabbitMQ, Redis, ZeroMQ, and ActiveMQ.
Kafka stores each topic as partitions that are divided into sequential segment files containing paired .log data and .index files, using global offsets and sparse memory‑mapped indexes to enable fast offset‑based lookups, efficient deletions, and minimal disk I/O in real‑world deployments.
This interview with Alibaba’s DRDS expert Shen Xun reveals how the distributed relational database service evolved from an internal tool to a cloud‑native solution, detailing its architecture, scalability advantages, middleware ecosystem, and real‑world performance during massive events like Double‑11.
This article surveys the architecture of long‑connection push systems for internet apps, distinguishes connection types, outlines client and server handling strategies, reviews mainstream backend solutions, and shares practical insights on security, API design, operations, queue services, debugging, and NoSQL usage.
This article reviews common notification mechanisms—signal, socket, FIFO, pipe, and eventfd/signalfd—used in high‑performance, lock‑free shared‑memory message queues, comparing their advantages, drawbacks, and suitability for different workload scenarios, and explains why kernel involvement can affect performance and when to prefer each method.
