This article explains how to guarantee that insert messages are processed before update messages for the same data identifier across different Kafka topics by applying fine‑grained distributed locks and a caching strategy to detect and correct out‑of‑order consumption.
This article explains why message queues are used, outlines common scenarios such as decoupling, asynchronous processing and traffic shaping, compares major MQ products, and provides practical guidance on high availability, preventing loss, duplicate consumption, ordering, backlog handling, and expiration.
This article explains how to guarantee that messages are never lost when using MQ middleware such as Kafka, RabbitMQ, or RocketMQ, outlines the key interview points, and provides practical design patterns, detection mechanisms, idempotency, and scaling strategies for reliable message delivery.
This article explains the typical problems encountered with message queues, such as message loss, duplicate delivery, and distributed transaction handling, and details various solutions including local message tables, MQ‑based transactions, and the specific mechanisms used by RocketMQ, Kafka, and RabbitMQ to ensure reliability and consistency.
This article presents Meituan's large‑scale Kafka deployment, describing the current state and challenges of massive data ingestion, detailing latency‑reduction techniques, cluster‑level optimizations, SSD‑based caching, isolation strategies, full‑link monitoring, lifecycle management, and future directions for high availability.
This article surveys various delayed queue implementations—including Kafka, RocketMQ, Redis (Redisson), and Netty’s HashedWheelTimer—examining their design principles, advantages, drawbacks, and integration strategies, and proposes a unified micro‑service architecture leveraging Kafka topics, Redis ZSETs, and thread‑pool optimizations for reliable message scheduling.
DTLE 4.22.01.0, an open‑source MySQL‑focused data transfer component, has been released, introducing Oracle‑MySQL incremental DDL/DML sync, enhanced WHERE clause support, initial Kafka schema‑change handling, and numerous bug fixes including binlog relay issues, character‑set handling, and CREATE TABLE … LIKE renaming problems.
The article details the architecture, key components, testing types, and operational results of Autohome's PTS platform, which uses Docker Swarm, gRPC, JMeter, Flume‑Kafka, and Flink to conduct large‑scale distributed load testing for the 818 event and outlines future improvements toward Kubernetes and direct Kafka logging.
This article details ByteHouse’s evolution from ClickHouse, presenting two real‑time analytics use cases, the technical selection process, performance bottlenecks such as write throughput and Kafka consumption, and the engineered solutions—including asynchronous indexing, multi‑threaded Kafka engines, and enhanced Buffer engines—that enable reliable, high‑throughput data processing at massive scale.
This article explains Kafka's timing wheel algorithm, detailing its O(1) delay operation design, multi‑layer wheel structure, and how Kafka advances time using a DelayQueue, while comparing it with Netty and other open‑source projects.
This article compiles a comprehensive list of common production problems encountered with Apache Flink, covering cluster sizing, checkpoint failures, backpressure analysis, resource allocation, deployment errors, UDF definitions, data skew, Kafka configurations, and provides detailed troubleshooting steps and best‑practice recommendations.
This article surveys delayed‑queue implementations, comparing Kafka, RocketMQ, and Redis (Redisson) designs, outlining their architectures, pros and cons, integration details, timing‑wheel mechanisms, and practical considerations for building a reliable distributed delayed‑queue service.
This article explores Kafka’s performance architecture, covering network and disk bottlenecks, sequential writes, zero‑copy techniques, page cache usage, Reactor‑based networking, batch processing, compression, partition concurrency, and file structures, providing practical optimization methods for high‑throughput streaming applications.
This article explains what an APM system is, explores the three core components—Logs, Traces, and Metrics—and details the design and implementation of the Dog APM platform, covering client data models, automatic instrumentation, server processing pipelines, and storage strategies.
This article presents a detailed interview-style walkthrough covering Hadoop cluster setup, HDFS components, MapReduce workflow, YARN advantages, Spark fundamentals, Kafka replication, Hive table types, and related big‑data concepts, providing concise explanations and practical insights for data engineers.
This article explains why message queues are used, outlines core interview questions, analyzes interviewers' expectations, compares major MQ products, and provides practical guidance on selecting the right queue for different scenarios.
This article explains the difference between synchronous and asynchronous microservice communication, outlines the benefits of async messaging, and compares three popular message brokers—RabbitMQ, Kafka, and Redis—based on scale, persistence, and consumer patterns, offering guidance on selecting the right broker for various use cases.
This article explains Kafka’s fundamental design concepts—including topics, partitions, replicas, consumer groups, and its network architecture—while highlighting performance features such as sequential writes, zero‑copy, log segmentation, and how the controller coordinates with ZooKeeper, providing a comprehensive overview for backend developers.
This article explains the design goals, exactly‑once semantics, Java transaction API, coordinator and log architecture, and practical performance trade‑offs of Apache Kafka's transactional messaging, helping developers build reliable stream‑processing applications.
The Smart Matching Engine for Haolo’s ride‑sharing service ingests driver and passenger orders via Kafka‑Flink pipelines into Elasticsearch, then applies multi‑stage matching—nearby search, itinerary‑based filtering using ETA, angle, distance, route‑similarity and shared‑mileage calculations—and finally ranks results with evolving pre‑sorting and algorithmic models, including PMML and deep‑learning, to optimize driver‑passenger pairing.
This article provides a comprehensive overview of Kafka, covering its layered architecture with Zookeeper, core concepts such as topics, partitions and consumer groups, communication workflow, partition selection strategies, rebalancing mechanisms, reliability configurations, replica synchronization, and reasons for moving away from Zookeeper, all explained in clear English.
This article reviews the background, architecture, and a series of performance‑optimizing techniques—including consumption, batch, storage, and execution‑engine tweaks—applied to a real‑time pipeline that processes hundreds of billions of records daily, and presents the resulting resource savings and latency improvements.
This article details the evolution from a Node.js‑based WebSocket gateway to a Go‑implemented, gRPC‑driven architecture, describing the redesign of connection handling, TLS off‑loading, socket ID generation, session management, custom Kafka headers, code‑level optimizations, and extensive performance testing that validates the new gateway’s scalability and resource efficiency.
This article describes the architectural evolution from Gateway 1.0 to Gateway 2.0 for a high‑traffic document collaboration platform, detailing the redesign of the WebSocket layer, resource‑usage optimizations, heartbeat mechanisms, custom Kafka headers, message serialization, load‑testing results, and the operational lessons learned to support millions of concurrent connections.
This article details the redesign of Shimo Docs' WebSocket gateway—from a Node.js/Socket.IO 1.0 version to a Go‑based 2.0 architecture—covering handshake degradation, TLS memory savings, SnowFlake SocketID generation, Redis‑based session broadcasting, heartbeat tuning, custom Kafka headers, object pooling, MessagePack compression, extensive performance testing, and the resulting stability and scalability improvements for handling half‑a‑million concurrent connections.
This article details the design, refactoring, and performance testing of a high‑traffic WebSocket gateway for Shimo Docs, covering the evolution from a Node.js Socket.IO version to a Go‑based microservice architecture, TLS memory tuning, socket ID generation, heartbeat handling, custom Kafka headers, and resource‑efficient scaling to half‑a‑million concurrent connections.
This article details the evolution from a Node.js Socket.IO gateway to a Go‑based, gRPC‑enabled WebSocket gateway that supports millions of concurrent connections, covering architecture redesign, TLS handling, socket ID generation, session management, heartbeat optimization, custom Kafka headers, code‑level refinements, and extensive performance testing results.
This article introduces the fundamentals of message queues and compares five popular products—ActiveMQ, RabbitMQ, ZeroMQ, Kafka, and RocketMQ—detailing their features, typical use cases, and providing curated learning resources for each.
Qunar’s DBA team overhauled their MySQL monitoring and alert system—originally built on Nagios and NRPE—by integrating a Kafka‑based pipeline, a custom alarm service, and MySQL‑stored alert templates, achieving flexible thresholds, granular silencing, high‑availability processing, and early‑stage intelligent management of alerts, slow queries, and disk space.
This article presents a detailed learning roadmap for aspiring big‑data experts, covering foundational programming languages, data structures, Linux basics, databases, distributed system theory, and essential frameworks such as Hadoop, Spark, Flink, Kafka, and provides curated B‑site video links and reference materials.
This article demystifies message queues by comparing them to parcel lockers, explains their role in asynchronous processing and system decoupling, details point‑to‑point and publish‑subscribe models, introduces JMS, AMQP, RabbitMQ, ActiveMQ, RocketMQ, and Kafka, and outlines their architectures and high‑availability mechanisms.
This article examines Martin Kleppmann’s view of Kafka as a database, breaking down how Kafka handles atomicity, consistency, isolation, and durability through event streams and partitioning, and argues why MySQL still remains essential for many workloads.
This article explains Kafka’s core design concepts—including the role of a message system, topics, partitions, producers, consumers, consumer groups, replica management, controller coordination with Zookeeper, performance optimizations like sequential writes and zero‑copy, and its network thread model—illustrated with diagrams and code snippets.
This article provides a comprehensive overview of popular Kafka monitoring solutions—including JMX, Kafka Manager (CMAK), Kafka Eagle, and Logi‑KafkaManager—detailing their features, installation steps, configuration examples, and comparative advantages, while also mentioning custom setups using JMXTrans, InfluxDB, and Grafana.
This article explains Kafka’s core architecture—including topics, partitions, replication, consumer groups, controller coordination with Zookeeper, and performance tricks like sequential writes and zero‑copy—to show how it achieves high‑throughput, fault‑tolerant messaging for large‑scale systems.
This article provides a visual walkthrough of Kafka's fundamental concepts—including its basic architecture, producer‑consumer model, topics, partitions, and cluster design—offering clear explanations and diagrams to help readers grasp how Kafka enables reliable, scalable message streaming in modern systems.
This article outlines HuoLaLa's journey from a fragmented multi‑cloud environment to a unified, platform‑driven database governance system, detailing the technical challenges, architectural decisions, key components for MySQL, Redis, Kafka, and other middleware, and the measurable stability and cost improvements achieved after a year of operation.
This comprehensive guide explains Kafka's core value for decoupling and asynchronous processing, details its producer‑consumer model, cluster architecture, log segmentation, zero‑copy I/O, hardware sizing, network planning, operational commands, throughput tuning, exception handling, consumer group mechanics, offset management, rebalance strategies, and internal mechanisms such as LEO/HW, controllers, delayed tasks, and time‑wheel scheduling.
This article explains Kafka's evolution—highlighting the introduction of Kafka Streams, the shift to a full distributed stream processing platform, practical learning paths, source‑code reading tips, common pitfalls, and the major new features introduced in Kafka 3.0.
This article explains the design of a distributed timer service using a time‑wheel data structure stored in Redis, covering application scenarios, required features, architecture components such as access layer, scheduler, worker, and management center, and detailing reliability and performance techniques.
This article compares RabbitMQ and Kafka by explaining their architectures, queue consumption and production mechanisms, highlighting the master‑mirror design of RabbitMQ versus Kafka's partitioned approach, and summarizing their throughput characteristics to help engineers choose the appropriate messaging middleware for their needs.
This article explains the fundamentals of message queues, their role in asynchronous processing and system decoupling, compares point‑to‑point and publish‑subscribe patterns, introduces Java Message Service (JMS) and its API, and outlines popular implementations such as ActiveMQ, RabbitMQ, RocketMQ, and Kafka with their architectures and high‑availability mechanisms.
This article compares Pulsar with Kafka, RocketMQ and InLong‑TubeMQ, focusing on cloud‑native multi‑tenant design, multi‑level flow control, compute‑storage separation, partitioning strategies, metadata storage, and message delivery mechanisms, highlighting each system's architectural advantages and trade‑offs.
Pulsar’s cloud‑native, multi‑tenant architecture with hierarchical flow control, stateless brokers separated from BookKeeper storage, global partitions supporting multiple consumers, push‑based dispatch, and precise acknowledgment outperforms Kafka, RocketMQ, and InLong‑TubeMQ, though its reliance on ZooKeeper and higher resource usage demand careful operational planning.
This guide shows how to build a real‑time UV and PV analytics pipeline on Tencent Cloud Oceanus by connecting a self‑hosted Kafka cluster to Flink SQL, using Redis for deduplicated visitor counts, page view logs, and conversion‑rate calculations via hop windows.
This article explains how to design and implement a centralized log monitoring platform using the ELK stack, Filebeat, Elastic APM, Prometheus, and Kafka Streams to collect, filter, visualize, and alert on petabyte‑level logs across thousands of microservices.
The article examines Kafka’s high‑availability mechanisms, detailing its multi‑replica design, ISR synchronization, leader election, and the critical role of the __consumer_offset topic, and explains why a single broker outage can render the entire cluster unusable unless replication factors are properly configured.
This article explains the fundamentals of message queues, describing how MQ, Kafka, and RocketMQ handle asynchronous processing, traffic shaping, and decoupling through topics, partitions, consumer groups, and storage mechanisms such as sequential writes and page‑cache optimization.
This guide walks through setting up a complete log‑collection and visualization pipeline—including SpringBoot log4j2 configuration, Kafka broker creation, Filebeat forwarding, Logstash processing, and Kibana dashboard setup—so you can capture, ship, and analyze application logs in real time.
The article examines Apache Kafka's strengths and shortcomings, explores the operational complexities of managing a Kafka deployment, and encourages organizations to reassess its value versus emerging alternatives by weighing maturity, scalability, and total cost of ownership.
This article explains the fundamentals of message queues, compares JMS and various MQ implementations such as RabbitMQ and Kafka, describes their architectures, usage scenarios, and key concepts like producers, consumers, topics, partitions, and high‑availability mechanisms.
This guide walks you through setting up a complete log‑collection and visualization pipeline—preparing servers, configuring a SpringBoot application with Log4j2, routing logs to Kafka, harvesting them with Filebeat, processing with Logstash, and finally visualizing in Elasticsearch and Kibana.
Apache Kafka 3.0 introduces major enhancements including KRaft consensus, deprecation of Java 8 and Scala 2.12 support, stronger producer guarantees, updated APIs, improved Kafka Connect, MirrorMaker 2 flexibility, and numerous KIP-driven feature upgrades, marking a significant step forward for the distributed streaming platform.
This article provides a comprehensive comparison between RocketMQ and Kafka, covering data reliability, performance, queue capacity, real‑time delivery, retry mechanisms, ordering guarantees, scheduled messages, transactional support, query capabilities, message tracing, consumer parallelism, filtering, and commercial backing, helping engineers choose the right messaging middleware for their workloads.
This article introduces the concept of message queues, explains their role in decoupling systems and improving performance, compares point‑to‑point and publish/subscribe models, and provides an overview of JMS, RabbitMQ, AMQP, and Kafka with practical examples and diagrams.
This article provides a comprehensive technical analysis of Kafka’s Controller component, covering its background, core responsibilities, data storage, election process, version‑specific improvements, monitoring techniques, and key source‑code excerpts to help engineers understand and manage Kafka clusters effectively.
This article explains Kafka's high‑availability architecture, covering multi‑replica redundancy, ISR synchronization, producer ACK settings, and the critical role of the __consumer_offset topic, and shows how to configure replication factors to prevent a single‑node outage from stopping consumption.
This article details the design and implementation of a real‑time, TB‑scale bill‑detail query platform that leverages Kafka for streaming, Debezium and Confluent Platform for change capture, Kudu for low‑latency storage, and Presto/Kylin for fast OLAP queries, while outlining deployment, integration, and future enhancements.
This article details the requirements, architecture, and implementation of a real-time and offline integrated query system, covering data ingestion via Debezium and Confluent Platform, storage in Kudu and HDFS, query engines Presto and Kylin, and strategies for data synchronization, partitioning, and scaling.
This article introduces the concepts, core features, deployment environments, architectures, high‑availability solutions, advantages and disadvantages of four widely used open‑source distributed message queue systems—Kafka, ActiveMQ, RabbitMQ, and RocketMQ—and provides a comparative analysis to help practitioners choose the right middleware for their distributed applications.
This article compares the architectures of RabbitMQ and Kafka, explaining their queue models, master‑mirror and partition mechanisms, performance trade‑offs, and provides guidance on choosing between them based on throughput requirements and system design considerations.
This article explains how Wix engineers built a robust, event‑driven messaging infrastructure on Kafka that serves over 1,400 microservices, detailing six key patterns—including consumption‑projection, end‑to‑end event flows, in‑memory KV stores, scheduling, transactional events, and aggregation—that improve scalability, resilience, and operational simplicity.
This article provides a comprehensive interview‑oriented overview of message queues, covering their use cases, popular frameworks, selection criteria, message models, reliability techniques, ordering, throughput optimization, high availability, and common Kafka interview questions.
This guide walks through setting up a complete log‑collection and visualization pipeline—preparing servers, configuring a SpringBoot project with Log4j2, deploying Kafka, installing Filebeat, creating Logstash pipelines, and visualizing logs in Elasticsearch and Kibana—so you can monitor application logs in real time.
The interview with Kuaishou senior architect Zhao Jianbo details the three‑phase evolution of its trillion‑scale big data platform, covering foundational Hadoop services, real‑time and OLAP extensions, deep customizations, Spring Festival Gala challenges, scheduling innovations, Hadoop usage, and the relationship between big data and cloud architectures.
This article explains Kafka's storage architecture, including segment files and indexing, and details the reliability mechanisms such as ISR, OSR, LEO, HW, producer acknowledgment levels, and leader election strategies to ensure data consistency and availability.
This tutorial walks through the end‑to‑end setup of a logging pipeline that starts with a SpringBoot application using Log4j2, forwards logs to Kafka, collects them with Filebeat, processes them via Logstash, and finally visualizes them in Elasticsearch and Kibana, covering server preparation, configuration files, and essential code snippets.
This article explains how DMA and zero‑copy techniques reduce the four memory copies and context switches typical of Linux I/O, detailing their mechanisms, implementations such as sendfile, mmap and Direct I/O, and real‑world usage in Kafka and MySQL to boost performance.
The article compares how RocketMQ, Apache Kafka, and Apache Pulsar implement transactional messaging—each using a two‑phase commit with half‑messages or transaction buffers, distinct coordinators, and idempotent producers—to provide atomicity and either strong consistency (RocketMQ) or exactly‑once delivery for high‑throughput stream processing (Kafka and Pulsar).
This article details Qunar's design and implementation of a data synchronization platform that aggregates MySQL data into Elasticsearch, supporting parallel ES5.x/7.x clusters, hot‑swap upgrades, flexible scaling, and end‑to‑end consistency with high availability mechanisms.
This article explains the costly four‑copy, four‑context‑switch data path in traditional Linux I/O, introduces DMA as a co‑processor that offloads memory transfers, describes zero‑copy techniques such as sendfile, mmap and Direct I/O, and shows how Kafka and MySQL leverage these methods to reduce CPU overhead and improve throughput.
This article describes how Ctrip built a data lineage system for its big data platform, covering the concept of data lineage, collection methods, open‑source tools such as Apache Atlas and DataHub, the in‑house table‑level and field‑level solutions, implementation details for Hive, Spark and Presto, storage in JanusGraph, and practical applications in data governance, metadata management, scheduling and sensitivity labeling.
An in‑depth look at Kafka’s high‑availability mechanisms reveals how multi‑replica design, ISR leader election, and the request.required.acks setting interact, why a single broker failure—especially of the __consumer_offset topic—can halt consumption, and how to configure replication factors to prevent such outages.
This article analyzes common like‑system business scenarios, derives functional requirements, proposes three architectural solutions (Redis‑based priority queue, relational database, and a hybrid cache‑MQ‑DB design), presents detailed implementation code, and evaluates each approach with extensive performance tests to guide practical backend design choices.
This article explains why Redis can be slow when used for delayed processing, shows how to implement a delayed queue with Redis ZSET, compares its advantages, and reviews alternative solutions such as RabbitMQ, RocketMQ, Kafka, Netty, and Java's DelayQueue.
Facing over a million concurrent connected vehicles, Chinese IoT leader ZhongRui chose RocketMQ over Kafka, leveraging Alibaba Cloud’s managed service to achieve low‑latency, high‑throughput, fault‑tolerant messaging, while adopting cloud‑native microservices, containerization, and serverless techniques to streamline operations and reduce costs.
ByteDance’s ByteHouse, an enterprise‑grade ClickHouse, powers real‑time recommendation and ad‑delivery analytics at massive scale, detailing two case studies, technical selections, architectural designs, and performance optimizations such as asynchronous indexing, multi‑threaded Kafka consumption, and enhanced buffer engines to ensure data integrity.
This tutorial walks through preparing servers, configuring a SpringBoot project with Log4j2, setting up Kafka, Filebeat, and Logstash to collect and parse logs, and finally visualizing them in Elasticsearch and Kibana, providing a full end‑to‑end logging solution.
Learn how to set up Kafka on CentOS, create and manage topics, adjust partitions, produce and consume messages using command‑line tools and Java code, and explore related utilities such as consumer groups, log inspection, and configuration tips for remote access.
This guide explains how to collect container logs with Kafka, deploy OpenFunction to process them serverlessly, write a Go handler that detects 404 errors, and send alerts to Slack, demonstrating a cost‑effective, scalable log‑processing solution on Kubernetes.
This article explains why a message system like Kafka is crucial for decoupling services, handling asynchronous workflows such as e‑commerce flash sales, controlling traffic, and achieving high concurrency, high availability, and high performance through sequential disk writes, zero‑copy reads, replication, and careful resource planning.
This article explains how TiCDC, a change‑data‑capture tool for TiDB, addresses incremental extraction, cross‑region hot‑standby, and stream processing needs, outlines its architecture, discusses early‑version issues, and provides best‑practice recommendations for stable, high‑performance data synchronization.
This article explains how Kafka achieves remarkable speed and massive throughput by using sequential disk I/O, OS page cache, zero‑copy transfers, partitioned log segments with indexes, batch processing, and efficient compression, making it a cornerstone of modern big‑data pipelines.
This article explains the complete Kafka topic deletion workflow, from the client’s deleteTopics request through Zookeeper node creation, controller coordination, broker StopReplica handling, log renaming, delayed file removal, and final cleanup, while providing code excerpts and practical Q&A for common pitfalls.
This article explains how 360 Commercialization implements cross‑IDC hot‑standby for Kafka using MirrorMaker 2, covering MM2 fundamentals, architecture, internal topics, deployment on Kubernetes, design goals, solution details, challenges such as dynamic configuration and offset reverse‑mapping, and productized risk mitigation.
This article compares RabbitMQ and Kafka by explaining their architectures, queue consumption and production mechanisms, highlighting RabbitMQ's single‑master limitation and Kafka's partition‑based design for higher throughput, and provides guidance on selecting the appropriate middleware based on performance needs.
When a Kafka cluster reaches load limits or experiences sudden traffic spikes, urgent expansion is needed, but data migration can be time‑consuming and risky; this guide outlines several practical techniques—including adjusting retention, adding partitions, leader switching, and single‑replica operation—to quickly scale clusters while minimizing data movement and service disruption.
The article explains message queues, their primary use cases—async processing, system decoupling, and traffic peak shaving—compares five mainstream products (ActiveMQ, RabbitMQ, Kafka, RocketMQ, Pulsar), and examines core design principles such as architecture, storage choices, push/pull models, and consumer groups, highlighting broader relevance to distributed systems.
This brief article explains the importance of reading Kafka's source code for technical interviews, outlines its simple module structure with emphasis on the core module, and highlights critical areas such as offset handling, storage, leader‑follower synchronization, and connector integration with Spark and Flink.
This article explains the fundamentals of message queues, compares point-to-point and publish‑subscribe models, introduces Java Message Service (JMS) and its programming model, and provides an overview of popular MQ implementations such as RabbitMQ, ActiveMQ, RocketMQ, and Kafka, highlighting their architectures and use cases.
This step‑by‑step guide shows how to configure a Spring Boot application with Log4j2, ship logs via Filebeat to Kafka, process them with Logstash, and visualize the data in Elasticsearch and Kibana, covering server setup, Maven dependencies, configuration files, and deployment commands.
This article explains the need for a messaging system, explores Kafka's core concepts, cluster architecture, performance optimizations like sequential disk writes and zero‑copy, and provides detailed guidance on sizing hardware, configuring producers and consumers, and managing a production Kafka deployment.
This guide walks through setting up Kafka Streams with Spring Boot 2.3, covering environment configuration, core concepts, topology design, and multiple practical examples—including message sending, listening, transformations, aggregations, filtering, branching, and multi‑field grouping—complete with full code snippets and execution results.
This article explains the fundamentals of message queues, compares JMS with popular implementations such as RabbitMQ and Kafka, describes their architectures, usage patterns, and how they improve system performance, scalability, and decoupling in backend applications.
This article provides a comprehensive introduction to Apache Kafka, covering its core concepts, terminology, data flow, configuration options such as compression and partitioning, producer and consumer behavior, replication, consumer groups, offset management, and log retention strategies.
This article explains Kafka's high‑availability mechanisms—including multi‑replica design, ISR synchronization, and the impact of the request.required.acks setting—while diagnosing why a single‑replica __consumer_offset topic can cause the entire consumer group to stop when a broker fails.
An in‑depth look at Kafka’s multi‑replica architecture, ISR mechanism, and ack settings reveals why a single broker failure—especially of the __consumer_offset topic—can halt consumer reads, and how proper replication factors and configuration can ensure true high‑availability.
This article compares RabbitMQ and Apache Kafka, explaining their core architectures, exchange and routing mechanisms, delivery guarantees, push vs pull models, dead‑letter handling, partitioning, consumer groups, log compaction, and the strengths and limitations of each system for scalable, ordered message processing.
This article details JD's end‑to‑end design of a real‑time browsing record platform that captures, stores, and queries up to 200 recent items per user with millisecond latency, covering architecture, hot‑cold data separation, microservice APIs, and streaming pipelines using Kafka, Flink, Jimdb, and HBase.
This article provides a comprehensive technical overview of Apache Kafka, covering its distributed messaging architecture, key features such as high‑throughput read/write, replication, partitioning, consumer group mechanics, offset management, rebalance processes, and practical code examples for synchronous and asynchronous offset commits.
