This article compares RabbitMQ, Kafka, and RocketMQ from a high‑availability perspective, explaining each system's deployment modes, replication mechanisms, and the trade‑offs between scalability and fault tolerance.
This article compares Redis list, Kafka, and Pulsar as messaging middleware, explaining their architectures, strengths, and weaknesses—including queue fundamentals, partitioning, cursor management, consumer groups, high‑availability mechanisms, storage strategies, and consumption models—to help readers choose the right solution for large‑scale systems.
This article explains how Kafka 2.8 removes its dependency on Zookeeper, describes Kafka's core concepts and its interaction with Zookeeper, outlines the role of the Controller, discusses operational complexities and upgrade paths with KIP‑500, and highlights the benefits of the new KRaft‑based architecture.
Kafka 2.8 replaces the external ZooKeeper dependency with an internal Quorum controller (KRaft), enabling ZooKeeper‑free deployments that reduce resource usage, improve performance, support larger clusters, but currently lack some features such as ACLs, transactions, and partition reassignment, making it unsuitable for production yet.
Kafka partitions are the fundamental storage units that enable topics to scale horizontally, maintain message order within each partition, provide fault‑tolerance through replication, and support parallel consumption, with various write strategies such as key‑based, round‑robin, and custom rules.
The article explains the design philosophy behind Kafka 2.8’s removal of Zookeeper, reviews Zookeeper’s classic leader‑election use cases, highlights its limitations, and shows how the Raft protocol provides a decentralized alternative for high‑availability leader selection in distributed messaging systems.
Kafka 2.8.0 introduces a major improvement by implementing the Raft consensus algorithm, allowing the platform to operate without ZooKeeper, which reduces system complexity and improves efficiency, though the ZooKeeper‑less mode is still early‑stage and not recommended for production environments.
This article provides an in‑depth, English‑language explanation of Kafka's overall architecture, including the roles of producers, consumers, topics, partitions, replication, Zookeeper coordination, controller election, and the NIO‑based network model, helping readers understand both concepts and practical configuration implications.
This article presents Kuaishou's comprehensive engineering practices for improving Flink's stability, task startup latency, and SQL performance, including high‑availability Kafka connectors, fault‑recovery mechanisms, I/O reductions, asynchronous job upgrades, aggregation optimizations, and future resource‑utilization plans.
This article provides a detailed overview of the ELK stack—including Elasticsearch, Logstash, Kibana, and Beats—explaining its components, why to use it for centralized log management, various deployment architectures, system tuning, security setup, and step‑by‑step installation and configuration commands for a production‑grade environment.
Kafka 2.8.0, released on April 19, 2021, introduces the groundbreaking Raft Metadata mode that eliminates the need for ZooKeeper, alongside numerous new features, bug fixes, and enhancements such as API controls for stream threads, SASL_SSL mutual TLS, and IP rate limiting.
This article compares the fundamentals and limitations of Redis list‑based queues with Kafka and Pulsar, explaining how partitions, cursors, replication, segment storage, and consumption models overcome Redis’s drawbacks and enable high‑performance, scalable, and highly available messaging systems.
This article compares the fundamentals and design trade‑offs of popular message‑queue middleware—Redis list, Kafka, and Pulsar—explaining their data structures, partitioning, persistence, consumer models, high‑availability mechanisms, and scalability challenges for developers and architects.
This article provides a visual walkthrough of Kafka's fundamental concepts—including producers, consumers, topics, partitions, and cluster architecture—illustrated with diagrams to help readers clearly understand how messages flow and are stored in a distributed streaming system.
This article explains the role of message queues in handling traffic spikes, decoupling services, and ensuring reliability, compares popular MQ solutions such as RabbitMQ, Kafka, RocketMQ and ActiveMQ, and discusses their architectures, advantages, drawbacks, idempotency, ordering, high‑availability and scaling strategies.
This article describes a real-time log processing platform that integrates Kafka, Flink, Drools rule engine, Redis, and Elasticsearch to unify heterogeneous log formats, extract business metrics, and provide configurable, dynamic data processing for large‑scale logging scenarios.
This comprehensive guide covers core distributed system theories, CAP consistency, transaction protocols, Redis features and persistence, Zookeeper coordination, message queue fundamentals, Kafka performance tricks, and practical solutions for caching, locking, and high‑concurrency scenarios.
This article details a financial company's exploration of Apache Flink for real‑time processing, covering its unique business constraints, end‑to‑end data pipeline, single‑table and multi‑table use cases, implementation challenges, code snippets, data initialization, testing strategies, and lessons learned.
The article compares basic in‑memory queues, Redis lists and streams, Kafka’s partitioned log architecture, and Pulsar’s compute‑storage separation, explaining each system’s core mechanisms, strengths, and limitations so readers can choose the most suitable message‑queue solution for their workloads.
This article explains Kafka’s three delivery semantics—At most once, At least once, and Exactly once—from both producer and consumer perspectives, details the required configuration settings, and discusses how Kafka ensures idempotence, transaction support, and prevents data loss, duplication, and ordering issues.
To meet iQIYI video production’s thousands‑QPS, petabyte‑scale, frequently‑updated data and large‑table join requirements, the team built a Spark‑plus‑ClickHouse real‑time warehouse that streams Kafka changes, joins HBase dimensions, and writes to ClickHouse, reducing reporting development time from days to hours while supporting both offline and real‑time analytics.
This article provides a detailed side‑by‑side analysis of four major distributed message‑queue systems—Kafka, RabbitMQ, RocketMQ, and ActiveMQ—examining them across seventeen criteria such as documentation, language support, protocols, storage, transactions, load balancing, clustering, management UI, availability, duplication, throughput, subscription models, ordering, acknowledgments, replay, retry, and concurrency.
This article provides a comprehensive overview of Kafka, covering its role as a distributed messaging middleware, core concepts, architecture components, common interview questions, command‑line tools, producer and consumer configurations, high‑availability mechanisms, delivery semantics, and performance optimizations for backend developers.
The DTLE 3.21.03.0 release introduces new MySQL data‑transfer configurations, performance improvements for Kafka full copies, and several bug fixes, while providing repository, documentation, and download instructions for users seeking a reliable open‑source replication component.
This article recounts the author's two‑year experience with Kafka in a high‑traffic restaurant ordering system, detailing why message ordering matters, the pitfalls of synchronous retries, message backlog, partition routing, primary‑key conflicts, database replication lag, and practical mitigation strategies for reliable backend processing.
This article shares a two‑year journey of using Kafka in a high‑traffic restaurant ordering system, covering why message order matters, how network glitches and partition routing cause failures, and the practical retry, partition‑balancing, and database strategies that finally eliminated backlog and duplication issues.
A Kafka broker outage can halt consumer consumption despite remaining brokers, due to replication settings, ISR mechanics, and the internal __consumer_offsets topic’s default replication factor, which this article explains and resolves with practical configuration steps.
This comprehensive guide introduces Kafka as a high‑throughput, distributed, publish‑subscribe messaging system, detailing its core concepts, architecture, features, replication, log management, reliability guarantees, and typical use cases such as log collection, real‑time analytics, and cross‑cluster mirroring.
iQIYI transformed its data infrastructure from a traditional offline T+1 model to a comprehensive real‑time ecosystem—leveraging Kafka, Flink, a three‑layer Stream Data Service Platform, the Talos drag‑and‑drop pipeline, and a Druid‑based analytics platform—to enable low‑latency monitoring, personalized recommendations, ad targeting, and continuous machine‑learning workflows while planning future stream‑batch integration and lake‑warehouse convergence.
This article explains how the creator red‑packet feature was engineered to guarantee idempotent, fault‑tolerant, and high‑throughput red‑packet claims by using multi‑level caching, empty placeholders, binlog‑driven synchronization, active cache invalidation, ordered Kafka consumption, and fallback strategies to resolve cache‑DB consistency issues.
This article explains how Uber built a multi‑region Kafka infrastructure with disaster‑recovery capabilities, detailing its replication topology, active/active and active/passive consumption modes, offset‑management service, and the challenges of ensuring reliable, low‑latency data streaming across regions.
This article explains why Kafka can lose messages at the broker, producer, and consumer levels, describes the underlying mechanisms such as page cache flushing and acknowledgment settings, and provides practical code examples and mitigation strategies to improve reliability.
This article explains how a growing e‑commerce platform replaced a heavyweight MySQL intermediate table with a binlog‑driven pipeline that streams changes to Elasticsearch via Kafka, detailing the architecture, modules, customizations, monitoring, and performance results.
This article explains how Apache Kafka is used for asynchronous microservice communication, identifies the common pitfall of consumer message failures, and evaluates retry‑topic patterns, their drawbacks, and alternative approaches such as back‑off retries and hidden topics while preserving message ordering and data consistency.
This article details ByteDance's evolution from offline ClickHouse ingestion to a robust real‑time analytics pipeline, covering external transaction handling, risks of direct INSERTs, recommendation and ad‑delivery use cases, Kafka Engine design, multi‑threaded consumption, fault‑tolerance improvements, platform tooling, and future roadmap.
This article explains the three places where Kafka can lose messages—Broker, Producer, and Consumer—detailing the underlying mechanisms, the impact of flush and ack settings, and practical configuration and coding strategies to minimize data loss.
This article explains how NetEase games collect heterogeneous logs, design a Flink‑driven streaming ETL pipeline, handle schema‑free sources, implement Python UDFs with Jython, optimize HDFS writes, manage real‑time and offline warehouses, and share practical tuning and fault‑tolerance techniques.
Uber operates the world’s largest Kafka cluster, handling trillions of messages daily, and has engineered a multi‑region deployment with active/active and active/passive consumption modes, offset management, and uReplicator to ensure high‑availability and seamless disaster recovery across data centers.
This case study explains how a client with a 2‑billion‑row MySQL table avoided disk exhaustion and poor query performance by streaming data through Kafka to Google BigQuery, partitioning the data, and cleaning it to reclaim storage—all without downtime.
This article explains how to build a highly scalable, cost‑effective game data collection pipeline using Serverless function compute, Kafka, and cloud services, covering architecture design, function implementation, deployment with Fun, Kafka configuration, and performance testing to handle massive traffic spikes.
This article compares ActiveMQ, RocketMQ, and Kafka across multiple dimensions such as messaging models, API completeness, language support, throughput, latency, availability, message loss risk, documentation, community activity, and commercial backing, helping readers choose the most suitable queue for their backend needs.
This comprehensive guide explains Kafka's role as a message system, detailing topics, partitions, producers, consumers, replication, controller, ZooKeeper coordination, performance optimizations like sequential writes and zero‑copy, and practical recommendations for hardware, configuration, and cluster deployment.
This article describes how JD's Energy Management Platform leverages ClickHouse as a high‑performance, MPP‑based OLAP engine to provide real‑time, multi‑dimensional analytics on IoT energy data, covering business background, technology selection, system architecture, data ingestion, storage, replication, and a generic query interface with code examples.
This article examines a Kafka cluster failure where partition 34 could not elect a leader due to index file corruption, explains the underlying sanity‑check logic, reproduces the fault, and offers practical recovery steps and configuration recommendations to prevent data loss.
This article walks through a real‑world Kafka incident where a sudden surge in client connections caused ISR churn, frequent connection drops, and high response time, and explains how monitoring thread idle rates and increasing network and I/O thread counts restored stability.
The DTLE 3.21.01.0 release introduces Kafka batch‑send parameters, support for rename‑table DDL, empty transactions for skipped DDL/DCL, and several bug fixes, while providing repository links, documentation, and installation tips for this MySQL‑focused data‑transfer component.
This tutorial walks through preparing three Kubernetes nodes, extracting and distributing Zookeeper, configuring its zoo.cfg and myid files, starting and verifying the Zookeeper ensemble, then installing Kafka, adjusting its server.properties, and finally launching Kafka across the cluster.
This article explains how Kafka producers decide which partition to send messages to, how topic partition counts are configured, and how consumer groups assign partitions to instances using default range and round‑robin strategies, with code examples for illustration.
This article compiles essential interview‑style questions and detailed answers on distributed system fundamentals—including CAP and BASE theories, consistency models, distributed transactions, Redis features and persistence, Zookeeper coordination, Kafka architecture, and common design patterns for high‑concurrency scenarios.
This article examines the technical challenges of scaling Kafka clusters to handle millions of partitions—including Zookeeper node explosion, replication overhead, controller recovery latency, and broker restart delays—and proposes solutions such as parallel ZK fetching, metadata synchronization via internal topics, logical cluster composition, and physical cluster splitting.
This guide explains Kafka's authentication and authorization mechanisms, covering SASL/PLAIN and SASL/SCRAM setups, JAAS file creation, server property configuration, ACL management, and provides complete Java producer and consumer examples for secure communication.
This article walks through designing and implementing a Go log‑collection agent that uses etcd for configuration storage, context for timeout and metadata handling, and Kafka for message consumption, complete with code examples, setup instructions, and a rate‑limiting utility.
This article presents a detailed case study of NetEase Cloud Music’s real‑time analytics platform built on Kafka and Flink, covering background, architectural choices, platform‑level design, operational challenges, solutions such as the Magina framework, and a Q&A on reliability and monitoring.
Didi’s Logi‑KafkaManager is a multi‑tenant Kafka cloud platform that consolidates dozens of clusters into a secure, isolated gateway‑driven service offering intuitive web‑based topic management, real‑time metrics visualization, automated diagnostics, quota governance and safe scaling, delivering high internal satisfaction and enterprise commercialization.
Meituan built an SSD‑based application‑layer cache for Kafka that bypasses PageCache contention between real‑time and delayed jobs, classifies log segments across SSD and HDD, limits flush rates, and achieves up to 80% latency reduction while guaranteeing stable real‑time consumption.
Yidun’s public-opinion monitoring platform transforms massive raw web data into a unified format by separating dynamic Groovy-script-driven cleaning from static processing, achieving real-time source integration, high throughput, scalability, and high availability while addressing format diversity, team coordination, and performance-flexibility trade-offs.
This article provides a detailed tutorial on integrating Kafka with Spring using Spring‑Kafka, covering simple setup, embedded Kafka testing, topic creation, message sending and receiving, transaction support, listener configurations, manual acknowledgment, error handling, retry and dead‑letter queues, and related code examples.
Facing massive, heterogeneous logs from numerous payment services, Ctrip’s data team designed a unified logging framework that extends log4j2, streams logs via Kafka to HDFS using a customized Camus pipeline, partitions and stores data in ORC for efficient Hive analysis, while addressing format, storage, and performance challenges.
This article explains how Kafka determines which partition a producer sends a record to, how partition counts are configured, and how consumer groups assign partitions using the default, range, and round‑robin strategies, complemented by detailed Java code examples.
This article breaks down Kafka’s network communication layer, detailing the roles of SocketServer, the Acceptor thread, Processor threads, and related classes such as RequestChannel, KafkaRequestHandlerPool, and key configuration parameters, while illustrating their interactions with diagrams.
The article explains how Twitter built the Account Activity Replay API by repurposing Kafka as a storage layer, detailing the system’s architecture, partitioning strategy, request handling, deduplication, and performance optimizations to provide reliable event recovery for developers.
The article explains Kafka's high‑availability architecture, covering multi‑replica redundancy, ISR mechanisms, producer acknowledgment settings, and a real‑world case where a broker crash halted consumption due to the __consumer_offsets topic's replication factor, then offers concrete remediation steps.
This article examines a Kafka outage scenario in a fintech company, explains Kafka’s multi-replica redundancy design, leader‑follower architecture, ISR mechanism, and how misconfiguration of the __consumer_offset topic can cause cluster-wide consumer failures, and provides solutions to ensure true high availability.
This article compares RabbitMQ and Apache Kafka by examining their internal designs, messaging models, ordering guarantees, routing, timing, retention, fault‑tolerance, scalability, and consumer complexity, then provides concrete guidance on when to choose each technology for real‑world systems.
This article explains the fundamentals of message‑queue middleware, outlines key scenarios such as asynchronous processing, application decoupling, traffic shaping, log handling and messaging, compares popular products like ActiveMQ, RabbitMQ, ZeroMQ and Kafka, and details JMS concepts and programming models.
This article explains how Kafka attains million‑level transactions per second by using sequential disk reads/writes, memory‑mapped files, DMA‑based zero‑copy transfers, and batch data transmission, detailing each technique and its impact on throughput and latency.
Kafka can sustain millions of transactions per second by writing data sequentially to disk, leveraging memory‑mapped files, employing zero‑copy DMA transfers, and batching messages, each technique reducing I/O overhead and CPU involvement, which together enable its high‑throughput performance in big‑data pipelines.
This article explains the leader election mechanisms used in big‑data systems—ZAB in Zookeeper, Raft’s role‑based election, their drawbacks such as split‑brain and ZooKeeper overload, and how Kafka’s controller‑based design solves these issues with efficient partition leader selection.
Kafka handles many requests that require asynchronous processing or waiting for conditions by attaching a timeout parameter; if the condition isn’t met within the timeout, Kafka returns a timeout response, and it implements this efficiently using a hierarchical Timing Wheel data structure that offers O(1) insertion and fast expiration checks.
This article provides a comprehensive walkthrough of Kafka's transaction mechanism, covering the transaction coordinator, producer initialization, partition handling, commit and abort processes, state management, high‑availability design, timeout handling, and relevant source code snippets.
This article explains Kafka’s internal index files—OffsetIndex and TimeIndex—including their file formats, how they store relative offsets and timestamps, the space‑saving optimizations, the processes for appending, truncating, and looking up entries, and best‑practice cautions for handling these indexes.
This article explains Kafka's index architecture, the AbstractIndex class hierarchy, how entry sizes are chosen, the use of memory‑mapped files, the binary‑search algorithm for locating index entries, and a cache‑friendly improvement that reduces page faults and I/O latency.
This article outlines five practical methods for synchronizing MySQL data to external systems such as Elasticsearch, Redis, and message queues, covering business‑layer hooks, middleware integration, scheduled tasks using updated_at, binlog parsing with ROW format, and handling mixed or statement binlog formats, plus open‑source tools.
This article chronicles the four‑year evolution of Chehaoduo Group’s Kafka ecosystem—from its initial role as a simple data‑ingestion layer to becoming the core of the company’s large‑scale data pipeline—detailing cluster management, upgrade strategies, multi‑cluster deployment, AVRO schema handling, SDK development, and operational lessons learned.
This article explains the advantages and disadvantages of using message middleware in microservice architectures and details practical solutions for handling concurrency, ordered processing, duplicate messages, and transactional messaging using patterns like partitioning, outbox tables, CDC, and RocketMQ's two‑phase commit.
The article thoroughly explains Kafka’s message‑queue reliability design and implementation, covering use‑case scenarios, core concepts, storage format, producer acknowledgment settings, broker replication mechanisms (ISR, HW, LEO), consumer delivery semantics, the epoch solution for synchronization, and practical configuration guidelines for various consistency and availability requirements.
This article examines Apache Pulsar, an open‑source messaging platform created by Yahoo, compares it with Kafka by outlining Kafka’s common pain points, highlights Pulsar’s multi‑tenant architecture, layered storage, built‑in functions, and security features, and discusses the trade‑offs of each solution.
This article details the three‑stage evolution of 58.com’s commercial data warehouse, describing its massive scale, four‑layer architecture, technical challenges, migrations from MapReduce to Hive and Flink, real‑time streaming upgrades, and the resulting improvements in stability, accuracy, and timeliness.
This article walks through the evolution of big‑data systems to modern stream processing, explains core Flink concepts such as state, checkpoints, event‑time and windowing, and details Kuaishou’s real‑time UV calculation and fast‑failover techniques for high‑availability streaming jobs.
The article explains why a single Kafka broker failure can render the whole cluster unavailable, detailing Kafka's multi‑replica architecture, ISR mechanism, leader election, producer acknowledgment settings, and the special handling required for the __consumer_offset topic.
The PHP Kafka client library supports PHP‑FPM and Swoole environments, implements all 50 Kafka APIs with compression, SSL, and SASL features, requires PHP ≥ 7.1 and Kafka ≥ 1.0.0, and can be installed via Composer.
This article details why the e‑commerce platform built its own Corgi message queue, the operational and cost drawbacks that prompted a move to Tencent Cloud CKafka, and the three‑phase migration strategy—including dual‑write, cut‑read, and cut‑write—while preserving message safety and low latency.
This article explains how to identify and resolve performance bottlenecks in Apache Flume by configuring Taildir sources, optimizing channel capacities, tuning Kafka sinks, adjusting JVM options, and using simple monitoring scripts, enabling a single Flume‑NG agent to sustain over 50,000 RPS in production.
This article outlines a request log analysis system that records core request fields, adds proxy‑related data, derives IP‑based ASN and geographic information, parses user‑agent details, and provides comprehensive metrics such as PV/QPS, UV, traffic, latency, status monitoring, and business‑specific insights, all visualized via an ELK‑Kafka architecture.
This article explains how to design a scalable log collection system for big‑data platforms by combining Flume for data ingestion, Kafka for buffering and high‑throughput transport, and downstream processing components, providing configuration examples and best‑practice recommendations.
This article presents NetEase Cloud Music's real‑time computing platform built on Flink and Kafka, covering background, architectural design, Kafka and Flink selection reasons, platformization, warehouse usage, encountered challenges, and the solutions implemented to improve reliability and performance.
This article reviews the background, requirements, and architectural designs of major open‑source log collection systems—including Facebook’s Scribe, Apache’s Chukwa, LinkedIn’s Kafka, Cloudera’s Flume—and evaluates mature monitoring tools such as ELK, highlighting their features, use cases, advantages, and drawbacks for large‑scale log processing.
This article provides a comprehensive overview of Kafka as a distributed messaging middleware, covering its core concepts, architecture, producer and consumer mechanics, common interview questions, configuration options, high‑availability guarantees, and performance optimizations for backend developers.
This article explains the zero‑copy data transfer technique, compares it with traditional read/write approaches, shows Java NIO code examples, and discusses its use in high‑performance systems such as Kafka and Spark, highlighting the reductions in context switches and memory copies.
This guide walks through building a secure, production‑grade logging pipeline by deploying an ELK stack (Elasticsearch, Logstash, Kibana) with X‑Pack security, a Kafka message queue with SASL authentication, and Filebeat agents, covering environment preparation, certificate generation, configuration files, and startup scripts.
The article explains how Kafka handles timeout‑based requests using a Timing Wheel data structure, detailing its design, parameters, operation principles, overflow handling, and providing Scala code examples that illustrate O(1) task insertion compared to traditional O(logN) delay queues.
This article explains why message queues are used, their advantages and disadvantages, compares Kafka, ActiveMQ, RabbitMQ, and RocketMQ, and provides interview-focused guidance on high availability, idempotency, reliability, ordering, and handling of message backlog.
This article explains why real‑time data warehouses are needed, outlines their core principles, compares them with offline warehouses, describes typical use cases such as real‑time OLAP, dashboards, feature generation and monitoring, and provides a step‑by‑step guide to designing, implementing, and operating a Flink‑based streaming warehouse with Kafka, HBase, and metadata management.
The article explains how Kafka implements a hierarchical time‑wheel—a ring of bucket sets with a map for O(1) insertion, removal and expiration—to replace costly per‑tick scans, enabling efficient management of millions of delayed tasks across multiple timeout layers.
This article explains how Redis can be used to implement delayed queues, compares its advantages with other solutions such as RabbitMQ, RocketMQ, Kafka, Netty and Java DelayQueue, and provides practical guidance on using sorted sets and timestamps for time‑based task scheduling.
This article explains the fundamental architecture of message middleware, its key roles such as peak shaving, asynchronous processing and decoupling, the two consumption models (publish‑subscribe and point‑to‑point), and introduces core Kafka concepts with practical Java code examples.
This guide provides a step‑by‑step tutorial on downloading, installing, configuring, and starting Alibaba Canal and its adapter to achieve real‑time incremental data synchronization from MySQL binlog to destinations such as Kafka, including code snippets and configuration details.
This article details the architecture, protocol design, permission control, reliability mechanisms, and scaling strategies of a long‑connection gateway built with OpenResty, Kafka, and Redis, illustrating how to share persistent connections across multiple business services while ensuring high performance and fault tolerance.
This guide explains what Canal is, its key features and limitations, the underlying binlog replication principle, and provides detailed, step‑by‑step instructions for downloading, configuring, and launching both Canal Server and Canal Adapter to achieve high‑performance real‑time data synchronization.
This article explains how Kafka consumer groups assign partitions, the four situations that trigger a rebalance, the impact of consumer poll timeouts, and practical ways to tune max.poll.interval.ms and max.poll.records to avoid rebalance‑related errors.
