BestHub
Sign in
Tagged articles
1273 articles
Page 8 of 13
Java Architect Essentials
Java Architect Essentials
Jul 23, 2021 · Backend Development

Preventing Message Loss in RabbitMQ and Kafka: Transactions, Confirm Mode, Persistence, and Configuration Best Practices

This article explains the common points where messages can be lost in RabbitMQ and Kafka, compares transaction and confirm mechanisms, describes how to enable persistence and proper acknowledgments, and provides concrete configuration recommendations for producers and consumers to ensure reliable message delivery.

KafkaMessage ReliabilityRabbitMQ
0 likes · 9 min read
Preventing Message Loss in RabbitMQ and Kafka: Transactions, Confirm Mode, Persistence, and Configuration Best Practices
360 Quality & Efficiency
360 Quality & Efficiency
Jul 23, 2021 · Big Data

Introduction to Apache Kafka: Architecture, APIs, Code Examples, and Optimization

This article provides a comprehensive overview of Apache Kafka, covering its definition, architecture, command‑line API usage, Java producer and consumer examples, core principles such as high availability and message ordering, configuration tuning, and a summary of its advantages as a high‑throughput, fault‑tolerant streaming platform.

ConfigurationConsumerKafka
0 likes · 12 min read
Introduction to Apache Kafka: Architecture, APIs, Code Examples, and Optimization
Java Interview Crash Guide
Java Interview Crash Guide
Jul 23, 2021 · Operations

How to Build a Scalable APM System: Inside the Dog Architecture

This article explains what an APM system is, compares logs, traces and metrics, reviews popular tools, and then details the design and implementation of the in‑house Dog APM platform—including client data models, Kafka pipelines, processing pipelines, storage in ClickHouse/Cassandra, and UI visualizations.

APMKafkaMetrics
0 likes · 28 min read
How to Build a Scalable APM System: Inside the Dog Architecture
Top Architect
Top Architect
Jul 22, 2021 · Backend Development

RabbitMQ vs Apache Kafka: In‑Depth Comparison and Guidance for Choosing the Right Messaging System

This article explains the fundamental differences between RabbitMQ and Apache Kafka, covering their architectures, message models, ordering guarantees, routing capabilities, timing controls, retention policies, fault‑tolerance mechanisms, scalability, and consumer complexity, and then provides practical criteria to help architects decide which solution best fits a given use case.

ComparisonKafkaMessage Queue
0 likes · 22 min read
RabbitMQ vs Apache Kafka: In‑Depth Comparison and Guidance for Choosing the Right Messaging System
Open Source Linux
Open Source Linux
Jul 18, 2021 · Operations

Why a Single Kafka Broker Failure Can Halt All Consumers – The HA Explained

This article explains Kafka's high‑availability mechanisms, covering multi‑replica design, ISR synchronization, leader election, acknowledgment settings, and the hidden risk of the __consumer_offset topic's single‑replica configuration that can cause an entire cluster to become unavailable when one broker fails.

Distributed SystemsISRKafka
0 likes · 9 min read
Why a Single Kafka Broker Failure Can Halt All Consumers – The HA Explained
Top Architect
Top Architect
Jul 16, 2021 · Fundamentals

Introduction to Message Queues, JMS, MQ, and Kafka

This article provides a comprehensive overview of message queues, explaining their purpose, usage scenarios, two communication models, and detailing Java Message Service (JMS) as well as popular MQ implementations such as RabbitMQ and Kafka, complete with diagrams and code examples.

JMSKafkaMessage Queue
0 likes · 15 min read
Introduction to Message Queues, JMS, MQ, and Kafka
Youzan Coder
Youzan Coder
Jul 15, 2021 · Backend Development

Message Queue Architecture Comparison: NSQ, Kafka, and RocketMQ in Distributed Systems

The article compares the architectures of NSQ (YouZan branch), Kafka, and RocketMQ—detailing their coordination mechanisms, storage models, consistency guarantees, and operational trade‑offs—while recommending Kafka for log‑big‑data workloads, RocketMQ for massive topic counts, and NSQ for extensibility and lightweight deployment.

Distributed SystemsKafkaMessage Queue
0 likes · 16 min read
Message Queue Architecture Comparison: NSQ, Kafka, and RocketMQ in Distributed Systems
IT Architects Alliance
IT Architects Alliance
Jul 12, 2021 · Big Data

Kafka Core Concepts, Architecture, Performance Optimization, and Operational Practices

This article explains Kafka's fundamental principles, cluster architecture, data performance techniques such as zero‑copy and log segmentation, resource planning for high‑throughput scenarios, and provides practical operational commands and custom partitioning examples for reliable, high‑availability deployments.

Cluster DeploymentKafkamessage queues
0 likes · 32 min read
Kafka Core Concepts, Architecture, Performance Optimization, and Operational Practices
DevOps
DevOps
Jul 12, 2021 · Operations

The First Four Chaos Experiments to Run on Apache Kafka

This article explains how to use chaos engineering with Gremlin to design, execute, and analyze four experiments that test Kafka broker load, message loss, split‑brain scenarios, and ZooKeeper outages, helping improve the reliability and resilience of Kafka deployments.

Distributed SystemsGremlinKafka
0 likes · 18 min read
The First Four Chaos Experiments to Run on Apache Kafka
Full-Stack Internet Architecture
Full-Stack Internet Architecture
Jul 10, 2021 · Backend Development

Why Use Message Queues? Benefits, Drawbacks, High Availability, Idempotency, and Practical Tips

The article explains why message queues are essential for decoupling systems, improving latency, handling traffic spikes, ensuring high availability, maintaining order, and achieving idempotent consumption, while also discussing their disadvantages, configuration details for Kafka, RabbitMQ, RocketMQ, and practical troubleshooting strategies.

IdempotencyKafkaMQ
0 likes · 23 min read
Why Use Message Queues? Benefits, Drawbacks, High Availability, Idempotency, and Practical Tips
Spring Full-Stack Practical Cases
Spring Full-Stack Practical Cases
Jul 7, 2021 · Backend Development

Master Kafka Topic Management & Message Flow on CentOS with Java

This guide walks through setting up Kafka on CentOS, creating and configuring topics, inspecting topic and consumer group details, adjusting partitions, and implementing Java producer and consumer applications, complete with command-line examples, code snippets, and troubleshooting tips for remote server access.

CentOSConsumerJava Producer
0 likes · 9 min read
Master Kafka Topic Management & Message Flow on CentOS with Java
Top Architect
Top Architect
Jul 6, 2021 · Backend Development

Interview Guide: Why Use Message Queues, Their Pros and Cons, and Comparison of Kafka, ActiveMQ, RabbitMQ, and RocketMQ

This article explains why message queues are used in system architecture, outlines their advantages such as decoupling, asynchronous processing, and traffic shaping, discusses their drawbacks, and compares four popular MQ solutions—Kafka, ActiveMQ, RabbitMQ, and RocketMQ—to help candidates ace interview questions.

Backend ArchitectureInterview PreparationKafka
0 likes · 12 min read
Interview Guide: Why Use Message Queues, Their Pros and Cons, and Comparison of Kafka, ActiveMQ, RabbitMQ, and RocketMQ
JD Retail Technology
JD Retail Technology
Jul 5, 2021 · Backend Development

Design and Implementation of JD's Real-Time Browsing Record System

The article describes JD's real-time browsing record system architecture, detailing its four modules—storage, query, real-time reporting, and offline reporting—along with hot‑cold data separation, use of Jimdb, HBase, Kafka, and Flink to achieve millisecond‑level latency and high throughput for billions of user records.

BrowsingFlinkHBase
0 likes · 12 min read
Design and Implementation of JD's Real-Time Browsing Record System
MaGe Linux Operations
MaGe Linux Operations
Jul 3, 2021 · Backend Development

How to Build a Go Log Collector with etcd, Context, and Kafka Integration

This article walks through redesigning a Go‑based log‑collection framework, introducing etcd for distributed configuration, demonstrating context for timeout and data propagation, and showing how to integrate Kafka consumers while improving concurrency handling and adding rate‑limiting mechanisms.

BackendKafkacontext
0 likes · 16 min read
How to Build a Go Log Collector with etcd, Context, and Kafka Integration
dbaplus Community
dbaplus Community
Jun 30, 2021 · Backend Development

Unlock Kafka’s Speed: Deep Dive into Performance Secrets and Optimizations

This article provides a comprehensive technical guide to Kafka performance, covering the core bottlenecks of network, disk and complexity, detailing optimization techniques such as concurrency, compression, batching, caching and algorithms, and explaining how Kafka’s sequential write, zero‑copy, page cache, reactor‑based network model, batch handling, partition concurrency, and file structure contribute to high throughput.

KafkaZero Copyjava
0 likes · 17 min read
Unlock Kafka’s Speed: Deep Dive into Performance Secrets and Optimizations
Efficient Ops
Efficient Ops
Jun 28, 2021 · Backend Development

Why a Single Kafka Broker Failure Stops All Consumers – Understanding HA

This article explains Kafka's high‑availability mechanisms, covering multi‑replica design, ISR synchronization, leader election, the impact of the request.required.acks setting, and how the default __consumer_offset topic can become a single point of failure, with concrete steps to fix it.

KafkaReplicationconsumer-offset
0 likes · 9 min read
Why a Single Kafka Broker Failure Stops All Consumers – Understanding HA
Tencent Cloud Middleware
Tencent Cloud Middleware
Jun 28, 2021 · Big Data

Getting Started with Kafka’s New KRaft Mode: A Step‑by‑Step Guide

This article introduces Apache Kafka’s KRaft (Kafka Raft) mode, explains its architectural differences from ZooKeeper‑based deployments, details essential configuration parameters, and provides a complete step‑by‑step procedure—including commands and utility tools—to set up and operate a KRaft cluster.

ConfigurationDeploymentDistributed Systems
0 likes · 14 min read
Getting Started with Kafka’s New KRaft Mode: A Step‑by‑Step Guide
DataFunTalk
DataFunTalk
Jun 26, 2021 · Big Data

Building a Scalable Big Data Service System at Didi: Practices and Lessons

Zhang Liang shares Didi's four-stage journey of constructing and governing large‑scale open‑source big‑data engine services—including engine selection, hardware sizing, PaaS platform building, proxy architecture, and governance—highlighting practical challenges, solutions, and ROI‑driven best practices for Kafka, Elasticsearch, Flink, and related technologies.

Big DataData InfrastructureElasticsearch
0 likes · 16 min read
Building a Scalable Big Data Service System at Didi: Practices and Lessons
Selected Java Interview Questions
Selected Java Interview Questions
Jun 25, 2021 · Backend Development

Message Queues for Interviews: Why Use MQ, Benefits, Drawbacks, and Comparison of Kafka, ActiveMQ, RabbitMQ, and RocketMQ

This article explains why message queues are used in modern systems, outlines common interview questions about MQ, discusses the advantages of decoupling, asynchronous processing, and traffic shaping, examines the pros and cons of MQ, and compares the four major MQ products—Kafka, ActiveMQ, RabbitMQ, and RocketMQ—to help candidates prepare for technical interviews.

ActiveMQKafkaMQ
0 likes · 11 min read
Message Queues for Interviews: Why Use MQ, Benefits, Drawbacks, and Comparison of Kafka, ActiveMQ, RabbitMQ, and RocketMQ
Java Interview Crash Guide
Java Interview Crash Guide
Jun 25, 2021 · Backend Development

Understanding Kafka Transactions: TC Service, Producer Flow, and Code Walkthrough

This article explains how Kafka implements transactions, detailing the role of the Transaction Coordinator (TC) service, the transaction flow diagram, producer initialization, partition handling, offset commits, commit and abort processes, and includes a complete Java code example with client‑side and server‑side components.

ConsumerKafkaProducer
0 likes · 22 min read
Understanding Kafka Transactions: TC Service, Producer Flow, and Code Walkthrough
DataFunTalk
DataFunTalk
Jun 21, 2021 · Big Data

Flink + Iceberg 0.11 Practices in Qunar Data Platform

This article shares Qunar's experience using Flink together with Apache Iceberg 0.11 to address real‑time data warehouse challenges, covering background pain points, Iceberg architecture, solutions for Kafka data loss and Hive latency, and optimization practices such as small‑file handling, sorting, and checkpoint management.

Big DataData LakeFlink
0 likes · 13 min read
Flink + Iceberg 0.11 Practices in Qunar Data Platform
Java Interview Crash Guide
Java Interview Crash Guide
Jun 21, 2021 · Backend Development

How to Prevent Message Loss, Duplicates, and Backlog in Distributed Queues

This article explains practical techniques for detecting lost messages, ensuring reliable delivery across production, storage, and consumption stages, handling duplicate deliveries with idempotent designs, managing message backlogs through performance tuning, and using transactional messages to achieve distributed transaction consistency.

IdempotencyKafkaMessage Loss
0 likes · 23 min read
How to Prevent Message Loss, Duplicates, and Backlog in Distributed Queues
Code Ape Tech Column
Code Ape Tech Column
Jun 21, 2021 · Operations

Why Simple Kafka Retries Fail and How to Build a Robust Message‑Failure Strategy

This article analyzes common Kafka consumer failure scenarios, explains why naïve retry‑topic or message‑skip approaches can break ordering and data consistency, and presents practical patterns—including error classification, in‑consumer backoff, hidden topics, and DLQ handling—to design resilient asynchronous microservice communication.

Dead Letter QueueError HandlingKafka
0 likes · 21 min read
Why Simple Kafka Retries Fail and How to Build a Robust Message‑Failure Strategy
IT Architects Alliance
IT Architects Alliance
Jun 20, 2021 · Backend Development

Kafka Architecture, Core Concepts, and Operational Best Practices

This article provides a comprehensive overview of Kafka's architecture, core concepts, high‑throughput design, replication, network model, capacity planning, producer and consumer tuning, custom partitioning, rebalance strategies, broker management, and operational tools for building and maintaining robust distributed messaging systems.

Kafkaperformance
0 likes · 29 min read
Kafka Architecture, Core Concepts, and Operational Best Practices
ITFLY8 Architecture Home
ITFLY8 Architecture Home
Jun 20, 2021 · Backend Development

How to Prevent Message Loss and Ensure Reliable Delivery in Distributed Systems

This article explains practical techniques for detecting lost messages, guaranteeing reliable production, storage, and consumption stages, handling duplicate deliveries with idempotent designs, managing message backlogs, and implementing distributed transactions using transactional messages in modern message queue systems.

IdempotencyKafkaReliability
0 likes · 18 min read
How to Prevent Message Loss and Ensure Reliable Delivery in Distributed Systems
Java Architect Essentials
Java Architect Essentials
Jun 15, 2021 · Big Data

Comprehensive Guide to Apache Kafka: Concepts, Installation, Configuration, and Usage

This article provides a thorough overview of Apache Kafka, covering its core streaming concepts, key components such as topics, partitions, producers and consumers, common use cases, step‑by‑step installation and multi‑broker configuration, fault‑tolerance testing, and an introduction to Kafka Connect for data import/export.

ConsumerDistributed StreamingInstallation
0 likes · 24 min read
Comprehensive Guide to Apache Kafka: Concepts, Installation, Configuration, and Usage
Programmer DD
Programmer DD
Jun 14, 2021 · Databases

Master Real‑Time Change Data Capture with Debezium and Spring Boot

Learn how to capture and stream real‑time database changes using Debezium’s distributed CDC framework, configure MySQL binlog, integrate the embedded engine with Spring Boot, and process change events with sample code and Docker setup for robust data pipelines.

CDCChange Data CaptureDebezium
0 likes · 11 min read
Master Real‑Time Change Data Capture with Debezium and Spring Boot
DeWu Technology
DeWu Technology
Jun 12, 2021 · Backend Development

Design and Optimization of a High‑Throughput Messaging Platform

To handle exploding daily traffic, the team rebuilt the messaging platform with a unified API, concurrent consumption, MongoDB storage, and a priority scheme that uses separate Kafka topics and adjustable pull ratios, while employing a state‑machine‑driven thread pool, multi‑tier delay mechanisms, and MongoDB/Redis‑based fatigue control, delivering fast, traceable, hierarchical urgent delivery with visual metrics and content safety.

KafkaMessagingdelay queue
0 likes · 5 min read
Design and Optimization of a High‑Throughput Messaging Platform
IT Architects Alliance
IT Architects Alliance
Jun 11, 2021 · Backend Development

Understanding Message Queues: From Redis List to Kafka and Pulsar

This article explains the evolution of message‑queue middleware by comparing the basic double‑ended queue implementation, Redis list usage, Kafka’s partitioned log architecture with segment files and sparse indexes, and Pulsar’s compute‑storage separation using BookKeeper, highlighting their designs, strengths, and trade‑offs.

Distributed SystemsKafkaMessage Queue
0 likes · 28 min read
Understanding Message Queues: From Redis List to Kafka and Pulsar
ITFLY8 Architecture Home
ITFLY8 Architecture Home
Jun 10, 2021 · Backend Development

Why Kafka Beats Redis List: A Deep Dive into Message Queue Architecture

This article compares popular message middleware such as Redis, Kafka, and Pulsar, explaining their underlying data structures, strengths and weaknesses, and how concepts like partitions, replication, cursors, and storage segmentation enable high performance, scalability, and reliability in modern distributed messaging systems.

BackendDistributed SystemsKafka
0 likes · 29 min read
Why Kafka Beats Redis List: A Deep Dive into Message Queue Architecture
IT Architects Alliance
IT Architects Alliance
Jun 5, 2021 · Big Data

How to Build a Real‑Time Recommendation System with Flink, HBase, and Docker

This article walks through a complete real‑time recommendation system built on Apache Flink, detailing its v2.0 architecture, modules for user behavior, interest, and product profiling, the recommendation algorithms (hot‑list, collaborative filtering, item similarity), and step‑by‑step Docker deployment of MySQL, Redis, HBase, and Kafka.

DockerFlinkHBase
0 likes · 11 min read
How to Build a Real‑Time Recommendation System with Flink, HBase, and Docker
MaGe Linux Operations
MaGe Linux Operations
Jun 3, 2021 · Big Data

Why Kafka Handles Billions of Messages: Architecture, Use Cases, and Fast Performance

This article introduces Kafka, LinkedIn’s high‑throughput distributed messaging system, explains its core concepts such as brokers, topics, partitions, offsets, producers, consumers, and consumer groups, outlines common use cases like asynchronous decoupling and data‑stream processing, and details its fast performance mechanisms, fault‑tolerance, installation, and configuration steps.

Big DataData StreamingInstallation
0 likes · 11 min read
Why Kafka Handles Billions of Messages: Architecture, Use Cases, and Fast Performance
Tencent Cloud Developer
Tencent Cloud Developer
May 27, 2021 · Big Data

An Introduction to Kafka: Architecture, Core Components, Service Governance, Performance Optimizations, and Installation Guide

Kafka is a high‑throughput distributed publish‑subscribe system that uses brokers, topics, partitions, offsets, producers, consumers, and Zookeeper for metadata and leader election, offering fast sequential disk writes, page‑cache zero‑copy transfers, ISR‑based replication, and includes step‑by‑step installation of JDK, Zookeeper, and Kafka.

Big DataDistributed MessagingInstallation
0 likes · 11 min read
An Introduction to Kafka: Architecture, Core Components, Service Governance, Performance Optimizations, and Installation Guide
Top Architect
Top Architect
May 22, 2021 · Big Data

Kafka Basics: Topics, Partitions, Producers, Consumers, and Cluster Architecture

This article provides a comprehensive introduction to Kafka, covering its role as a message system, core concepts such as topics, partitions, producers, consumers, messages, the cluster architecture with replicas and controllers, performance optimizations, log segmentation, and network design, all illustrated with diagrams and code examples.

Big DataKafkaMessage Queue
0 likes · 13 min read
Kafka Basics: Topics, Partitions, Producers, Consumers, and Cluster Architecture
IT Architects Alliance
IT Architects Alliance
May 22, 2021 · Big Data

Flink-Based Real‑Time Recommendation System: Architecture, Logic, and Docker Deployment Guide

This article presents a comprehensive walkthrough of a Flink‑powered recommendation system, detailing its v2.0 architecture, module functions, recommendation algorithms (hotness, product similarity, collaborative filtering), front‑end and back‑end UI, and step‑by‑step Docker deployment of MySQL, Redis, HBase, and Kafka services.

Big DataDockerFlink
0 likes · 11 min read
Flink-Based Real‑Time Recommendation System: Architecture, Logic, and Docker Deployment Guide
Dada Group Technology
Dada Group Technology
May 21, 2021 · Backend Development

Implementing Data Heterogeneity for JD Daojia Order Fulfillment: Architecture, Canal Integration, and Lessons Learned

This article examines JD Daojia's order fulfillment system, detailing the challenges of high‑volume prompt‑sound queries, the division of responsibilities among Redis, MySQL, and Elasticsearch, the adoption of Canal for asynchronous data replication, deployment practices with Kafka and Zookeeper, and the key operational lessons learned.

Backend ArchitectureCanalElasticsearch
0 likes · 14 min read
Implementing Data Heterogeneity for JD Daojia Order Fulfillment: Architecture, Canal Integration, and Lessons Learned
Architect
Architect
May 19, 2021 · Big Data

Flink-Based Real-Time Recommendation System Architecture and Deployment Guide

This article presents a comprehensive overview of a Flink-powered real-time recommendation system, detailing its v2.0 architecture, module functions, recommendation algorithms, front‑end and back‑end interfaces, Docker‑based deployment of MySQL, Redis, HBase, Kafka, and step‑by‑step startup procedures.

DockerFlinkHBase
0 likes · 9 min read
Flink-Based Real-Time Recommendation System Architecture and Deployment Guide
Full-Stack Internet Architecture
Full-Stack Internet Architecture
May 19, 2021 · Backend Development

Understanding Message Queues: Benefits, Design Challenges, and Transactional Solutions

This article explores the role of message queues in microservice architectures, discussing their advantages such as decoupling, asynchronous processing, and load shedding, while also addressing design challenges like concurrency, ordering, duplicate handling, and transactional messaging with solutions including Kafka partitions, outbox patterns, CDC, and RocketMQ.

CDCKafkaMessage Queue
0 likes · 12 min read
Understanding Message Queues: Benefits, Design Challenges, and Transactional Solutions
Architecture Digest
Architecture Digest
May 17, 2021 · Big Data

Technical Architecture Overview of Toutiao: Data Pipeline, User Modeling, Recommendation System, and Microservices

The article provides a comprehensive technical overview of Toutiao's rapid growth, detailing its massive user base, data collection and processing pipelines, user modeling, cold‑start strategies, recommendation engines, storage solutions, push notification mechanisms, and the underlying microservice and PaaS architecture.

Big DataHadoopKafka
0 likes · 8 min read
Technical Architecture Overview of Toutiao: Data Pipeline, User Modeling, Recommendation System, and Microservices
Java High-Performance Architecture
Java High-Performance Architecture
May 17, 2021 · Backend Development

How to Tackle Message Queue Backlogs and Prevent Data Loss

This article explains why message queues accumulate, the risks of discarded messages, disk exhaustion, and massive pending loads, and provides practical strategies—including avoiding TTL, using monitoring alerts, temporary queues, and partition scaling—to quickly recover and process backlogged messages.

BacklogConsumerKafka
0 likes · 5 min read
How to Tackle Message Queue Backlogs and Prevent Data Loss
Code Ape Tech Column
Code Ape Tech Column
May 17, 2021 · Backend Development

Ensuring Reliable Message Delivery and Idempotence in RabbitMQ and Kafka

This article explains common scenarios that cause message loss or non‑idempotent processing in RabbitMQ and Kafka, and presents practical solutions such as persistent delivery, confirm mechanisms, delayed delivery, and unique‑ID plus fingerprint strategies to achieve reliable and idempotent message transmission.

IdempotenceKafkaRabbitMQ
0 likes · 6 min read
Ensuring Reliable Message Delivery and Idempotence in RabbitMQ and Kafka
Java High-Performance Architecture
Java High-Performance Architecture
May 15, 2021 · Backend Development

How to Ensure Ordered Messaging with RabbitMQ and Kafka

This article explains how to achieve ordered message processing by coordinating both producers and consumers, covering the differences between RabbitMQ's simple queue ordering and Kafka's partition-based approach, and offering practical techniques for global and partial ordering.

ConsumerKafkaProducer
0 likes · 5 min read
How to Ensure Ordered Messaging with RabbitMQ and Kafka
MaGe Linux Operations
MaGe Linux Operations
May 14, 2021 · Big Data

Build a Billion-Scale ELK Logging Platform with Filebeat, Kafka, Elasticsearch

This guide walks through the complete architecture and step‑by‑step deployment of a billion‑scale ELK logging system, covering Filebeat agents, Kafka buffering, Logstash processing, Elasticsearch indexing, and Kibana visualization, including configuration files, version details, and best‑practice tips for scaling and security.

ELKElasticsearchFilebeat
0 likes · 12 min read
Build a Billion-Scale ELK Logging Platform with Filebeat, Kafka, Elasticsearch
Programmer DD
Programmer DD
May 14, 2021 · Backend Development

Why Simple Retries Fail in Kafka and How to Build Robust Failure Strategies

This article explains Kafka's core concepts, the challenges of consumer failures in microservice architectures, why naïve retry loops or message skipping are insufficient, and presents a nuanced approach that distinguishes recoverable from unrecoverable errors, using back‑off retries and hidden topics to preserve ordering and data integrity.

Distributed SystemsKafkaMicroservices
0 likes · 24 min read
Why Simple Retries Fail in Kafka and How to Build Robust Failure Strategies
MaGe Linux Operations
MaGe Linux Operations
May 12, 2021 · Big Data

Visualizing Kafka: Core Concepts Explained with Diagrams

This article visually breaks down Kafka’s core concepts—including producers, consumers, topics, partitions, consumer groups, and cluster architecture—explaining how messages flow, are stored, and replicated across partitions and nodes, while highlighting the role of ZooKeeper in managing metadata.

Distributed SystemsKafkaMessaging
0 likes · 5 min read
Visualizing Kafka: Core Concepts Explained with Diagrams
vivo Internet Technology
vivo Internet Technology
May 12, 2021 · Big Data

Kafka at Trillion-Scale: Ensuring High Availability, Performance, and Operational Best Practices

The article presents a comprehensive guide for running Kafka at trillion‑record daily traffic, detailing version upgrades, data migration, traffic throttling, monitoring, load balancing, resource isolation, security, disaster recovery, Linux tuning, platform automation, performance evaluation, future roadmap, and community contribution practices.

Kafkaperformance
0 likes · 34 min read
Kafka at Trillion-Scale: Ensuring High Availability, Performance, and Operational Best Practices
IT Architects Alliance
IT Architects Alliance
May 11, 2021 · Big Data

Demystifying Kafka: Core Concepts of Topics, Partitions, and Architecture

This article provides a clear, visual walkthrough of Kafka’s fundamental architecture, explaining how producers and consumers interact, the role of topics and partitions, consumer groups, and ZooKeeper’s coordination, helping readers grasp message flow, storage, ordering, and fault‑tolerance in a distributed streaming system.

KafkaMessage QueuePartition
0 likes · 6 min read
Demystifying Kafka: Core Concepts of Topics, Partitions, and Architecture
Code Ape Tech Column
Code Ape Tech Column
May 10, 2021 · Industry Insights

Why Kafka Beats Redis List: A Deep Dive into Modern Messaging Middleware

This article compares Redis list, Kafka, and Pulsar as messaging middleware, explaining their architectures, strengths, and weaknesses—including queue fundamentals, partitioning, cursor management, consumer groups, high‑availability mechanisms, storage strategies, and consumption models—to help readers choose the right solution for large‑scale systems.

Distributed SystemsKafkaMessaging
0 likes · 30 min read
Why Kafka Beats Redis List: A Deep Dive into Modern Messaging Middleware
macrozheng
macrozheng
May 8, 2021 · Big Data

Why Kafka 2.8 Drops Zookeeper: Architecture, Challenges, and KIP‑500

This article explains how Kafka 2.8 removes its dependency on Zookeeper, describes Kafka's core concepts and its interaction with Zookeeper, outlines the role of the Controller, discusses operational complexities and upgrade paths with KIP‑500, and highlights the benefits of the new KRaft‑based architecture.

Distributed SystemsKIP-500KRaft
0 likes · 10 min read
Why Kafka 2.8 Drops Zookeeper: Architecture, Challenges, and KIP‑500
Architect's Tech Stack
Architect's Tech Stack
May 7, 2021 · Backend Development

Kafka 2.8 Introduces KRaft: Removing ZooKeeper with an Internal Quorum Controller

Kafka 2.8 replaces the external ZooKeeper dependency with an internal Quorum controller (KRaft), enabling ZooKeeper‑free deployments that reduce resource usage, improve performance, support larger clusters, but currently lack some features such as ACLs, transactions, and partition reassignment, making it unsuitable for production yet.

KRaftKafkaMessage Queue
0 likes · 4 min read
Kafka 2.8 Introduces KRaft: Removing ZooKeeper with an Internal Quorum Controller
Sohu Tech Products
Sohu Tech Products
May 5, 2021 · Big Data

Kafka Architecture and Implementation Principles – Part 2

This article provides an in‑depth, English‑language explanation of Kafka's overall architecture, including the roles of producers, consumers, topics, partitions, replication, Zookeeper coordination, controller election, and the NIO‑based network model, helping readers understand both concepts and practical configuration implications.

KafkaZooKeeper
0 likes · 17 min read
Kafka Architecture and Implementation Principles – Part 2
DataFunTalk
DataFunTalk
May 2, 2021 · Big Data

Continuous Optimization and Practice of Flink at Kuaishou

This article presents Kuaishou's comprehensive engineering practices for improving Flink's stability, task startup latency, and SQL performance, including high‑availability Kafka connectors, fault‑recovery mechanisms, I/O reductions, asynchronous job upgrades, aggregation optimizations, and future resource‑utilization plans.

Big DataFlinkKafka
0 likes · 10 min read
Continuous Optimization and Practice of Flink at Kuaishou
IT Architects Alliance
IT Architects Alliance
May 1, 2021 · Big Data

Comprehensive Guide to ELK Stack (Elasticsearch, Logstash, Kibana) Installation, Configuration, and Architecture

This article provides a detailed overview of the ELK stack—including Elasticsearch, Logstash, Kibana, and Beats—explaining its components, why to use it for centralized log management, various deployment architectures, system tuning, security setup, and step‑by‑step installation and configuration commands for a production‑grade environment.

Big DataELKElasticsearch
0 likes · 22 min read
Comprehensive Guide to ELK Stack (Elasticsearch, Logstash, Kibana) Installation, Configuration, and Architecture
Programmer DD
Programmer DD
Apr 30, 2021 · Big Data

Kafka 2.8.0 Release: Say Goodbye to ZooKeeper with Raft Metadata Mode

Kafka 2.8.0, released on April 19, 2021, introduces the groundbreaking Raft Metadata mode that eliminates the need for ZooKeeper, alongside numerous new features, bug fixes, and enhancements such as API controls for stream threads, SASL_SSL mutual TLS, and IP rate limiting.

Big DataKafkaRaft
0 likes · 5 min read
Kafka 2.8.0 Release: Say Goodbye to ZooKeeper with Raft Metadata Mode
Architect
Architect
Apr 27, 2021 · Fundamentals

Understanding Message Queue Architectures: Redis List, Kafka, and Pulsar

This article compares the fundamentals and design trade‑offs of popular message‑queue middleware—Redis list, Kafka, and Pulsar—explaining their data structures, partitioning, persistence, consumer models, high‑availability mechanisms, and scalability challenges for developers and architects.

KafkaPulsarredis
0 likes · 28 min read
Understanding Message Queue Architectures: Redis List, Kafka, and Pulsar
Java High-Performance Architecture
Java High-Performance Architecture
Apr 26, 2021 · Fundamentals

Visualizing Kafka: Core Concepts Explained with Diagrams

This article provides a visual walkthrough of Kafka's fundamental concepts—including producers, consumers, topics, partitions, and cluster architecture—illustrated with diagrams to help readers clearly understand how messages flow and are stored in a distributed streaming system.

Distributed SystemsKafkaPartitions
0 likes · 6 min read
Visualizing Kafka: Core Concepts Explained with Diagrams
Intelligent Backend & Architecture
Intelligent Backend & Architecture
Apr 23, 2021 · Backend Development

Why Message Queues Are Essential: Benefits, Pitfalls, and Best Practices

This article explains the role of message queues in handling traffic spikes, decoupling services, and ensuring reliability, compares popular MQ solutions such as RabbitMQ, Kafka, RocketMQ and ActiveMQ, and discusses their architectures, advantages, drawbacks, idempotency, ordering, high‑availability and scaling strategies.

IdempotencyKafkaMQ
0 likes · 38 min read
Why Message Queues Are Essential: Benefits, Pitfalls, and Best Practices
IT Architects Alliance
IT Architects Alliance
Apr 20, 2021 · Big Data

Real-time Log Processing System Based on Flink and Drools

This article describes a real-time log processing platform that integrates Kafka, Flink, Drools rule engine, Redis, and Elasticsearch to unify heterogeneous log formats, extract business metrics, and provide configurable, dynamic data processing for large‑scale logging scenarios.

DroolsElasticsearchFlink
0 likes · 6 min read
Real-time Log Processing System Based on Flink and Drools
ITFLY8 Architecture Home
ITFLY8 Architecture Home
Apr 19, 2021 · Backend Development

Mastering Distributed Systems: CAP, Redis, Zookeeper, Kafka and More

This comprehensive guide covers core distributed system theories, CAP consistency, transaction protocols, Redis features and persistence, Zookeeper coordination, message queue fundamentals, Kafka performance tricks, and practical solutions for caching, locking, and high‑concurrency scenarios.

Distributed SystemsKafkaZooKeeper
0 likes · 34 min read
Mastering Distributed Systems: CAP, Redis, Zookeeper, Kafka and More
dbaplus Community
dbaplus Community
Apr 17, 2021 · Big Data

How a Traditional Finance Firm Tackles Real‑Time Analytics with Flink

This article details a financial company's exploration of Apache Flink for real‑time processing, covering its unique business constraints, end‑to‑end data pipeline, single‑table and multi‑table use cases, implementation challenges, code snippets, data initialization, testing strategies, and lessons learned.

FinancialFlinkHBase
0 likes · 13 min read
How a Traditional Finance Firm Tackles Real‑Time Analytics with Flink
Tencent Cloud Developer
Tencent Cloud Developer
Apr 12, 2021 · Backend Development

Understanding Message Queues: From Basic Queues to Redis, Kafka, and Pulsar

The article compares basic in‑memory queues, Redis lists and streams, Kafka’s partitioned log architecture, and Pulsar’s compute‑storage separation, explaining each system’s core mechanisms, strengths, and limitations so readers can choose the most suitable message‑queue solution for their workloads.

KafkaMessage QueuePulsar
0 likes · 29 min read
Understanding Message Queues: From Basic Queues to Redis, Kafka, and Pulsar
Spring Full-Stack Practical Cases
Spring Full-Stack Practical Cases
Apr 12, 2021 · Big Data

Master Kafka’s Delivery Guarantees: At‑Most, At‑Least, and Exactly‑Once Explained

This article explains Kafka’s three delivery semantics—At most once, At least once, and Exactly once—from both producer and consumer perspectives, details the required configuration settings, and discusses how Kafka ensures idempotence, transaction support, and prevents data loss, duplication, and ordering issues.

Delivery GuaranteesExactly-OnceIdempotence
0 likes · 23 min read
Master Kafka’s Delivery Guarantees: At‑Most, At‑Least, and Exactly‑Once Explained
iQIYI Technical Product Team
iQIYI Technical Product Team
Apr 9, 2021 · Big Data

Real-Time Data Warehouse at iQIYI Video Production Using Spark and ClickHouse

To meet iQIYI video production’s thousands‑QPS, petabyte‑scale, frequently‑updated data and large‑table join requirements, the team built a Spark‑plus‑ClickHouse real‑time warehouse that streams Kafka changes, joins HBase dimensions, and writes to ClickHouse, reducing reporting development time from days to hours while supporting both offline and real‑time analytics.

HBaseKafkaSpark
0 likes · 12 min read
Real-Time Data Warehouse at iQIYI Video Production Using Spark and ClickHouse
Code Ape Tech Column
Code Ape Tech Column
Apr 9, 2021 · Backend Development

Comprehensive Comparison of Kafka, RabbitMQ, RocketMQ, and ActiveMQ Across 17 Dimensions

This article provides a detailed side‑by‑side analysis of four major distributed message‑queue systems—Kafka, RabbitMQ, RocketMQ, and ActiveMQ—examining them across seventeen criteria such as documentation, language support, protocols, storage, transactions, load balancing, clustering, management UI, availability, duplication, throughput, subscription models, ordering, acknowledgments, replay, retry, and concurrency.

ActiveMQDistributed SystemsKafka
0 likes · 22 min read
Comprehensive Comparison of Kafka, RabbitMQ, RocketMQ, and ActiveMQ Across 17 Dimensions
Full-Stack Internet Architecture
Full-Stack Internet Architecture
Apr 8, 2021 · Backend Development

Kafka Interview Guide: Concepts, Architecture, Configuration, and Performance

This article provides a comprehensive overview of Kafka, covering its role as a distributed messaging middleware, core concepts, architecture components, common interview questions, command‑line tools, producer and consumer configurations, high‑availability mechanisms, delivery semantics, and performance optimizations for backend developers.

ConsumerDistributed MessagingKafka
0 likes · 20 min read
Kafka Interview Guide: Concepts, Architecture, Configuration, and Performance
Wukong Talks Architecture
Wukong Talks Architecture
Apr 3, 2021 · Backend Development

Two Years of Kafka in a Restaurant Order System: Problems, Solutions, and Lessons Learned

This article recounts the author's two‑year experience with Kafka in a high‑traffic restaurant ordering system, detailing why message ordering matters, the pitfalls of synchronous retries, message backlog, partition routing, primary‑key conflicts, database replication lag, and practical mitigation strategies for reliable backend processing.

Kafkadistributed-systemsmessage-queue
0 likes · 17 min read
Two Years of Kafka in a Restaurant Order System: Problems, Solutions, and Lessons Learned
dbaplus Community
dbaplus Community
Apr 1, 2021 · Backend Development

Avoid Kafka Pitfalls: Ensuring Message Order, Handling Retries, and Preventing Backlog

This article shares a two‑year journey of using Kafka in a high‑traffic restaurant ordering system, covering why message order matters, how network glitches and partition routing cause failures, and the practical retry, partition‑balancing, and database strategies that finally eliminated backlog and duplication issues.

KafkaMessage OrderingPartition Balancing
0 likes · 19 min read
Avoid Kafka Pitfalls: Ensuring Message Order, Handling Retries, and Preventing Backlog
Senior Brother's Insights
Senior Brother's Insights
Mar 29, 2021 · Backend Development

Why Does a Single Kafka Broker Failure Break All Consumers?

A Kafka broker outage can halt consumer consumption despite remaining brokers, due to replication settings, ISR mechanics, and the internal __consumer_offsets topic’s default replication factor, which this article explains and resolves with practical configuration steps.

ACKBackendConsumer Offsets
0 likes · 11 min read
Why Does a Single Kafka Broker Failure Break All Consumers?
Programmer DD
Programmer DD
Mar 29, 2021 · Big Data

Mastering Kafka: High‑Throughput Distributed Messaging Explained

This comprehensive guide introduces Kafka as a high‑throughput, distributed, publish‑subscribe messaging system, detailing its core concepts, architecture, features, replication, log management, reliability guarantees, and typical use cases such as log collection, real‑time analytics, and cross‑cluster mirroring.

Big DataDistributed MessagingKafka
0 likes · 15 min read
Mastering Kafka: High‑Throughput Distributed Messaging Explained
iQIYI Technical Product Team
iQIYI Technical Product Team
Mar 26, 2021 · Big Data

Evolution of iQIYI's Real-Time Big Data Ecosystem

iQIYI transformed its data infrastructure from a traditional offline T+1 model to a comprehensive real‑time ecosystem—leveraging Kafka, Flink, a three‑layer Stream Data Service Platform, the Talos drag‑and‑drop pipeline, and a Druid‑based analytics platform—to enable low‑latency monitoring, personalized recommendations, ad targeting, and continuous machine‑learning workflows while planning future stream‑batch integration and lake‑warehouse convergence.

AnalyticsBig DataFlink
0 likes · 13 min read
Evolution of iQIYI's Real-Time Big Data Ecosystem
Kuaishou Tech
Kuaishou Tech
Mar 25, 2021 · Backend Development

Designing a High‑Availability Cache Consistency Solution for the Creator Red Packet System

This article explains how the creator red‑packet feature was engineered to guarantee idempotent, fault‑tolerant, and high‑throughput red‑packet claims by using multi‑level caching, empty placeholders, binlog‑driven synchronization, active cache invalidation, ordered Kafka consumption, and fallback strategies to resolve cache‑DB consistency issues.

BinlogCache ConsistencyIdempotency
0 likes · 11 min read
Designing a High‑Availability Cache Consistency Solution for the Creator Red Packet System
Architecture Digest
Architecture Digest
Mar 25, 2021 · Big Data

Uber's Multi-Region Kafka Architecture and Disaster Recovery

This article explains how Uber built a multi‑region Kafka infrastructure with disaster‑recovery capabilities, detailing its replication topology, active/active and active/passive consumption modes, offset‑management service, and the challenges of ensuring reliable, low‑latency data streaming across regions.

Data StreamingKafkaOffset Management
0 likes · 9 min read
Uber's Multi-Region Kafka Architecture and Disaster Recovery
ITPUB
ITPUB
Mar 12, 2021 · Backend Development

How to Sync MySQL Data to Elasticsearch in Real-Time Using Binlog and Kafka

This article explains how a growing e‑commerce platform replaced a heavyweight MySQL intermediate table with a binlog‑driven pipeline that streams changes to Elasticsearch via Kafka, detailing the architecture, modules, customizations, monitoring, and performance results.

BackendBinlogElasticsearch
0 likes · 13 min read
How to Sync MySQL Data to Elasticsearch in Real-Time Using Binlog and Kafka
Top Architect
Top Architect
Mar 9, 2021 · Backend Development

Handling Kafka Consumer Failures and Retry Strategies in Microservices

This article explains how Apache Kafka is used for asynchronous microservice communication, identifies the common pitfall of consumer message failures, and evaluates retry‑topic patterns, their drawbacks, and alternative approaches such as back‑off retries and hidden topics while preserving message ordering and data consistency.

Consumer FailureKafkaMicroservices
0 likes · 23 min read
Handling Kafka Consumer Failures and Retry Strategies in Microservices
dbaplus Community
dbaplus Community
Mar 2, 2021 · Databases

How ByteDance Scaled Real‑Time Analytics with ClickHouse and Kafka Engine

This article details ByteDance's evolution from offline ClickHouse ingestion to a robust real‑time analytics pipeline, covering external transaction handling, risks of direct INSERTs, recommendation and ad‑delivery use cases, Kafka Engine design, multi‑threaded consumption, fault‑tolerance improvements, platform tooling, and future roadmap.

KafkaReal-time analyticsbackend-development
0 likes · 22 min read
How ByteDance Scaled Real‑Time Analytics with ClickHouse and Kafka Engine
Code Ape Tech Column
Code Ape Tech Column
Feb 26, 2021 · Backend Development

Why Kafka Messages Get Lost and How to Prevent It

This article explains the three places where Kafka can lose messages—Broker, Producer, and Consumer—detailing the underlying mechanisms, the impact of flush and ack settings, and practical configuration and coding strategies to minimize data loss.

Ack SettingsBrokerConfiguration
0 likes · 15 min read
Why Kafka Messages Get Lost and How to Prevent It
Programmer DD
Programmer DD
Feb 20, 2021 · Big Data

How Uber Built a Multi‑Region Kafka Architecture for Disaster Recovery

Uber operates the world’s largest Kafka cluster, handling trillions of messages daily, and has engineered a multi‑region deployment with active/active and active/passive consumption modes, offset management, and uReplicator to ensure high‑availability and seamless disaster recovery across data centers.

Active-ActiveActive-PassiveKafka
0 likes · 10 min read
How Uber Built a Multi‑Region Kafka Architecture for Disaster Recovery
Alibaba Cloud Native
Alibaba Cloud Native
Feb 8, 2021 · Big Data

How Serverless Architecture Supercharges Game Data Collection and Scaling

This article explains how to build a highly scalable, cost‑effective game data collection pipeline using Serverless function compute, Kafka, and cloud services, covering architecture design, function implementation, deployment with Fun, Kafka configuration, and performance testing to handle massive traffic spikes.

Big Data IngestionFunction ComputeKafka
0 likes · 22 min read
How Serverless Architecture Supercharges Game Data Collection and Scaling
Practical DevOps Architecture
Practical DevOps Architecture
Feb 8, 2021 · Backend Development

Comparison of Common Message Queues: ActiveMQ, RocketMQ, and Kafka

This article compares ActiveMQ, RocketMQ, and Kafka across multiple dimensions such as messaging models, API completeness, language support, throughput, latency, availability, message loss risk, documentation, community activity, and commercial backing, helping readers choose the most suitable queue for their backend needs.

ActiveMQBackendComparison
0 likes · 4 min read
Comparison of Common Message Queues: ActiveMQ, RocketMQ, and Kafka
Open Source Linux
Open Source Linux
Feb 7, 2021 · Big Data

Mastering Kafka: Core Concepts, Architecture, and High‑Performance Deployment

This comprehensive guide explains Kafka's role as a message system, detailing topics, partitions, producers, consumers, replication, controller, ZooKeeper coordination, performance optimizations like sequential writes and zero‑copy, and practical recommendations for hardware, configuration, and cluster deployment.

Big DataCluster DeploymentKafka
0 likes · 22 min read
Mastering Kafka: Core Concepts, Architecture, and High‑Performance Deployment