BestHub
Sign in
Tagged articles

Kafka

1415 articles · Page 4 of 15
Big Data Technology & Architecture
Big Data Technology & Architecture
Mar 24, 2025 · Big Data

Apache Kafka 4.0: Major New Features – KRaft Architecture, Consumer Group Protocol, Queue Mode, Java Upgrade, API Simplifications and More

Apache Kafka 4.0 introduces a groundbreaking KRaft architecture that removes ZooKeeper, a revamped consumer‑group protocol that dramatically speeds up rebalancing, a new queue mode for point‑to‑point messaging, upgraded Java requirements, streamlined APIs, and numerous performance and security enhancements, reshaping both development and operations for large‑scale streaming workloads.

Java 11Java 17KRaft
0 likes · 12 min read
Apache Kafka 4.0: Major New Features – KRaft Architecture, Consumer Group Protocol, Queue Mode, Java Upgrade, API Simplifications and More
Code Ape Tech Column
Code Ape Tech Column
Mar 14, 2025 · Backend Development

Eight Common Use Cases of Message Queues in Backend Development

This article explores eight common scenarios for using message queues in backend development, covering asynchronous processing, service decoupling, traffic shaping, delayed tasks, log aggregation, distributed transactions, remote calls, and broadcast notifications, each illustrated with Java, RocketMQ, and Kafka code examples.

JavaKafkaMQ
0 likes · 15 min read
Eight Common Use Cases of Message Queues in Backend Development
Java Tech Enthusiast
Java Tech Enthusiast
Mar 9, 2025 · Backend Development

Handling Duplicate Messages in Message Queues

Message queues can deliver duplicates under at‑least‑once semantics, so to protect idempotent business logic such as orders and payments you should combine producer‑side idempotence (e.g., Kafka’s enable.idempotence), broker‑side deduplication (e.g., Pulsar), and a consumer‑side guard using unique IDs stored in a database or Redis.

IdempotenceKafkaPulsar
0 likes · 7 min read
Handling Duplicate Messages in Message Queues
IT Services Circle
IT Services Circle
Mar 8, 2025 · Backend Development

Handling Duplicate Messages in Message Queues: Semantics, Producer and Broker Deduplication, and Consumer Strategies

Message queues can cause duplicate messages that affect idempotent business processes, so this article explains the three delivery semantics (At Least Once, Exactly Once, At Most Once), the causes of duplication, and practical deduplication techniques for producers, brokers (Kafka, Pulsar), and consumers using code examples.

IdempotenceKafkaPulsar
0 likes · 8 min read
Handling Duplicate Messages in Message Queues: Semantics, Producer and Broker Deduplication, and Consumer Strategies
Linux Ops Smart Journey
Linux Ops Smart Journey
Feb 27, 2025 · Information Security

Build Military‑Grade Kafka Security in Four Easy Steps

This guide walks you through four progressive stages—basic configuration, SSL encryption, SCRAM authentication, and combined SSL+SASL—showing how to harden Kafka with certificates, keystores, and client settings to achieve financial‑level protection.

Distributed MessagingKafkaSASL
0 likes · 9 min read
Build Military‑Grade Kafka Security in Four Easy Steps
dbaplus Community
dbaplus Community
Feb 26, 2025 · Operations

Choosing the Right Message Queue: A Deep Dive into Kafka, Pulsar, RocketMQ, RabbitMQ, and NSQ

This comprehensive guide explains the role of message queues in distributed systems, compares the architectures and core concepts of Kafka, Pulsar, RocketMQ, RabbitMQ, and NSQ, and provides practical criteria for selecting, configuring, and operating a message‑queue solution in high‑performance, high‑availability environments.

KafkaNSQPulsar
0 likes · 50 min read
Choosing the Right Message Queue: A Deep Dive into Kafka, Pulsar, RocketMQ, RabbitMQ, and NSQ
macrozheng
macrozheng
Feb 20, 2025 · Backend Development

Master Java Concurrency & SpringBoot: DJI Interview Insights & Advanced Locks

This article combines a DJI interview experience—including salary and bonus details—with an in‑depth guide to Java concurrency tools, lock implementations, SpringBoot AOP, filters vs. interceptors, Kafka ordering, zero‑copy techniques, and related backend development best practices.

Backend DevelopmentKafkaLocks
0 likes · 26 min read
Master Java Concurrency & SpringBoot: DJI Interview Insights & Advanced Locks
Top Architect
Top Architect
Jan 19, 2025 · Backend Development

Understanding Kafka Consumer: Offset Management, Rebalance, Partition Assignment, and Thread Safety

This article provides a comprehensive technical walkthrough of KafkaConsumer, covering Java configuration code, delivery semantics (at‑most‑once, at‑least‑once, exactly‑once), offset commit strategies, rebalance mechanisms, partition assignment algorithms, thread‑safety concerns, and internal poll implementation, followed by unrelated promotional content.

Backend DevelopmentConsumerKafka
0 likes · 16 min read
Understanding Kafka Consumer: Offset Management, Rebalance, Partition Assignment, and Thread Safety
IT Architects Alliance
IT Architects Alliance
Jan 9, 2025 · Backend Development

Understanding Message Queues: Comparing Kafka, RabbitMQ, and RocketMQ

Message queues act as asynchronous communication bridges in distributed systems, and this article examines three leading solutions—Kafka, RabbitMQ, and RocketMQ—detailing their performance, scalability, reliability, strengths, drawbacks, and suitable use‑cases to guide developers in selecting the right technology.

KafkaRabbitMQRocketMQ
0 likes · 13 min read
Understanding Message Queues: Comparing Kafka, RabbitMQ, and RocketMQ
dbaplus Community
dbaplus Community
Jan 5, 2025 · Big Data

How DeWu Halved Observability Costs Using AutoMQ and ClickHouse Storage‑Compute Separation

DeWu’s observability platform faced scalability, cost, and operational challenges from petabyte‑scale trace data, prompting a shift to a storage‑compute separated architecture that leverages AutoMQ’s Kafka‑compatible service and ClickHouse Enterprise’s SharedMergeTree engine, ultimately achieving up to 50% cost reduction and five‑fold cold‑read performance gains.

AutoMQClickHouseKafka
0 likes · 20 min read
How DeWu Halved Observability Costs Using AutoMQ and ClickHouse Storage‑Compute Separation
IT Services Circle
IT Services Circle
Jan 3, 2025 · Backend Development

Eight Common Use Cases of Message Queues (MQ) with Code Examples

This article explains eight typical scenarios for using message queues—including asynchronous processing, service decoupling, traffic shaping, delayed tasks, log collection, distributed transactions, remote invocation, and broadcast notifications—providing clear explanations and Java code snippets for each case.

Distributed TransactionsKafkaRocketMQ
0 likes · 13 min read
Eight Common Use Cases of Message Queues (MQ) with Code Examples
Ctrip Technology
Ctrip Technology
Jan 3, 2025 · Big Data

Design and Implementation of a Kafka Gatekeeper for FinOps Billing Data Quality Governance

This article describes the challenges of data quality in Ctrip’s hybrid‑cloud FinOps billing system and presents the design, implementation, and high‑availability deployment of a custom Kafka Gatekeeper proxy that performs pre‑validation, configurable rules, self‑service dashboards, and automated alerts to improve coverage, timeliness, and responsibility attribution.

Cloud NativeData QualityFinOps
0 likes · 17 min read
Design and Implementation of a Kafka Gatekeeper for FinOps Billing Data Quality Governance
Programmer XiaoFu
Programmer XiaoFu
Dec 26, 2024 · Backend Development

Avoid Misusing Spring Event: 6 Hard‑Earned Best Practices from a Real‑World Failure

The article recounts a production incident caused by improper use of Spring Event, explains why graceful shutdown, correct startup timing, suitable business scenarios, and reliability measures such as retries and idempotency are essential, and presents six concrete best‑practice recommendations to prevent similar failures.

Backend DevelopmentGraceful ShutdownKafka
0 likes · 11 min read
Avoid Misusing Spring Event: 6 Hard‑Earned Best Practices from a Real‑World Failure
Bin's Tech Cabin
Bin's Tech Cabin
Dec 25, 2024 · Backend Development

Unveiling Netty and Kafka Time Wheels: High‑Performance Scheduling Explained

This article explores the design and implementation of time wheel algorithms in Netty and Kafka, comparing their single‑layer and multi‑layer approaches, analyzing performance trade‑offs, and detailing how these systems achieve O(1) scheduling for massive delayed tasks while avoiding empty ticks.

BackendKafkaNetty
0 likes · 76 min read
Unveiling Netty and Kafka Time Wheels: High‑Performance Scheduling Explained
MaGe Linux Operations
MaGe Linux Operations
Dec 14, 2024 · Big Data

Master Kafka: From Core Concepts to Real-World Deployment

This comprehensive guide explains Kafka’s architecture, core APIs, topics and partitions, deployment steps, multi‑broker clustering, and practical use cases such as messaging, log aggregation, stream processing, and data import/export with Kafka Connect, providing a hands‑on tutorial for developers and engineers.

InstallationKafkaStreaming
0 likes · 30 min read
Master Kafka: From Core Concepts to Real-World Deployment
Top Architect
Top Architect
Dec 12, 2024 · Backend Development

Using Spring Transaction Hooks to Send Kafka Messages After Transaction Commit

This article explains how to leverage Spring's TransactionSynchronizationManager to detect active transactions, register synchronization callbacks, and asynchronously send Kafka messages only after a transaction successfully commits, illustrated with a payment‑system example and complete Java code snippets, while also noting thread‑local considerations and promotional offers.

JavaKafkaSpring
0 likes · 12 min read
Using Spring Transaction Hooks to Send Kafka Messages After Transaction Commit
JD Cloud Developers
JD Cloud Developers
Dec 11, 2024 · Operations

How to Prevent Message Queue Disorder in Distributed Systems

This article examines the causes of message queue disorder in distributed systems and presents practical solutions such as ordered messaging, pre‑consumption checks, state machines, and monitoring to improve system stability and data consistency.

KafkaRocketMQmessage ordering
0 likes · 9 min read
How to Prevent Message Queue Disorder in Distributed Systems
Alibaba Cloud Big Data AI Platform
Alibaba Cloud Big Data AI Platform
Dec 9, 2024 · Big Data

Why Kafka Falls Short for Real‑Time Analytics and How Fluss Changes the Game

Flink Forward Asia 2024 highlighted the limitations of Kafka for real‑time analytics—lack of updates, poor data exploration, costly back‑tracking, and high network overhead—while introducing Fluss, a columnar streaming storage that offers low‑latency reads, CDC, lake‑stream integration, and efficient Delta Join for scalable, fast analytics.

Delta JoinFlinkKafka
0 likes · 15 min read
Why Kafka Falls Short for Real‑Time Analytics and How Fluss Changes the Game
Xiaohongshu Tech REDtech
Xiaohongshu Tech REDtech
Dec 5, 2024 · Big Data

Interview with Jianchen: Journey from Open Source Contributor to Data Engineer at Xiaohongshu

In this interview, Xiaohongshu data engineer Jianchen recounts his evolution from a computer‑science student discovering open‑source through MIT6.824 to contributing to SOFAJRaft and Apache RocketMQ, detailing his OSPP projects, the decision to join Xiaohongshu, and his work on a cloud‑native Kafka engine that cut storage and compute usage by half.

Apache RocketMQCloud NativeKafka
0 likes · 11 min read
Interview with Jianchen: Journey from Open Source Contributor to Data Engineer at Xiaohongshu
Java Architect Essentials
Java Architect Essentials
Dec 4, 2024 · Backend Development

Spring Boot Project Initialization, Version Management, and Common Scaffold Code Guide

This article walks through initializing a Spring Boot project, handling version compatibility between Spring Cloud, Spring Boot, and Kafka, demonstrates common scaffolding code such as global exception handling, logging, CORS configuration, and recommends useful tools like Maven, embedded Redis, MyBatis‑Plus, and MapStruct for backend development.

Backend DevelopmentJavaKafka
0 likes · 9 min read
Spring Boot Project Initialization, Version Management, and Common Scaffold Code Guide
Sanyou's Java Diary
Sanyou's Java Diary
Dec 2, 2024 · Big Data

Understanding Kafka: Core Architecture, Storage, and Reliability Explained

This article provides a comprehensive overview of Kafka, covering its overall structure, key components such as brokers, producers, consumers, topics, partitions, replicas, leader‑follower mechanics, logical and physical storage models, producer and consumer workflows, configuration parameters, partition assignment strategies, rebalancing, log retention and compaction, indexing, zero‑copy transmission, and the reliability concepts that ensure data durability.

ArchitectureData StreamingKafka
0 likes · 18 min read
Understanding Kafka: Core Architecture, Storage, and Reliability Explained
Top Architect
Top Architect
Nov 25, 2024 · Backend Development

Using Spring Transaction Hooks to Send Kafka Messages After Transaction Commit

This article explains how to leverage Spring's TransactionSynchronizationManager to detect active transactions and register synchronization callbacks so that Kafka messages are sent asynchronously only after a transaction successfully commits, ensuring data consistency without impacting the main business flow.

JavaKafkaSpring
0 likes · 12 min read
Using Spring Transaction Hooks to Send Kafka Messages After Transaction Commit
Top Architect
Top Architect
Nov 20, 2024 · Big Data

Understanding Distributed Systems and Kafka: Architecture, Message Ordering, and Java Consumer Practices

This article explains the fundamentals of distributed systems, introduces Apache Kafka's architecture and components, discusses how Kafka ensures ordered message consumption, and provides Java consumer configuration tips to maintain message order, offering practical guidance for backend developers working with streaming data.

JavaKafkabig data
0 likes · 11 min read
Understanding Distributed Systems and Kafka: Architecture, Message Ordering, and Java Consumer Practices
Programmer1970
Programmer1970
Nov 18, 2024 · Backend Development

How to Resolve Message Backlog and Why Kafka Outperforms RocketMQ

The article outlines practical solutions for RocketMQ message backlog—including scaling consumers, optimizing logic, batch consumption, producer throttling, broker tuning, monitoring, and emergency plans—then explains why Kafka delivers higher performance through sendfile zero‑copy, simple log storage, efficient memory use, and batch sending, and finally compares the two systems on reliability, retry, transactions, filtering, ordering, and suitable use cases.

BacklogKafkaPerformance
0 likes · 8 min read
How to Resolve Message Backlog and Why Kafka Outperforms RocketMQ
Huolala Tech
Huolala Tech
Nov 14, 2024 · Operations

How Huolala Scaled Kafka: From Integrated Design to Cloud‑Native Elastic Architecture

This article chronicles the evolution of Huolala’s Kafka infrastructure—from an integrated compute‑storage design to a separated compute‑storage model with multi‑tenant deployment, and finally to a cloud‑native elastic architecture—detailing the challenges of capacity awareness, alarm configuration, and cost‑effective performance optimization.

KafkaOperationscapacity planning
0 likes · 9 min read
How Huolala Scaled Kafka: From Integrated Design to Cloud‑Native Elastic Architecture
Tencent Cloud Developer
Tencent Cloud Developer
Nov 14, 2024 · Cloud Native

ZooKeeper Core Knowledge and Typical Application Scenarios

Although many platforms are dropping ZooKeeper, this guide explains its CP‑oriented architecture, znode structure, watcher mechanism, Zab consensus, leader election, and common patterns such as publish/subscribe, load balancing, naming, master election, distributed locks and queues, giving architects essential fundamentals for coordination services.

Consensus AlgorithmDistributed CoordinationKafka
0 likes · 25 min read
ZooKeeper Core Knowledge and Typical Application Scenarios
macrozheng
macrozheng
Nov 12, 2024 · Backend Development

Boost Kafka Throughput in Spring Boot: Batch Consumption Guide

This article demonstrates how to integrate Kafka with Spring Boot, add necessary dependencies and configuration, implement both single‑message and batch consumers, and tune batch settings to dramatically improve processing speed for millions of records in a microservice environment.

BackendBatchProcessingJava
0 likes · 12 min read
Boost Kafka Throughput in Spring Boot: Batch Consumption Guide
JD Tech
JD Tech
Nov 7, 2024 · Big Data

JDQ Kafka Bandwidth Control and Rate Limiting Architecture

This article analyzes the limitations of native Apache Kafka rate‑limiting, introduces JDQ's multi‑dimensional, dynamic, priority‑aware bandwidth control framework, and demonstrates its effectiveness through partition‑level, single‑machine, and hierarchical elastic throttling experiments for large‑scale e‑commerce data streams.

Distributed StreamingElastic ThrottlingKafka
0 likes · 18 min read
JDQ Kafka Bandwidth Control and Rate Limiting Architecture
Sohu Tech Products
Sohu Tech Products
Nov 6, 2024 · Operations

Design and Implementation of a Business Operation Log Management System Using Canal and Elasticsearch

The article presents a decoupled business operation log management architecture that uses Alibaba’s Canal to capture MySQL binlog changes, streams them through Kafka, and stores structured before‑and‑after records in Elasticsearch with nested mappings, enabling multi‑table correlation via transaction IDs, visual querying, and reliable rollback without modifying application code.

BackendCanalElasticsearch
0 likes · 12 min read
Design and Implementation of a Business Operation Log Management System Using Canal and Elasticsearch
Mingyi World Elasticsearch
Mingyi World Elasticsearch
Nov 3, 2024 · Big Data

How to Build a Scalable Business Operation Log System with Canal and Elasticsearch

This article walks through the design and implementation of a decoupled, high‑performance business operation log solution that captures MySQL binlog changes via Canal, streams them through Kafka, and stores and queries them in Elasticsearch, addressing challenges such as batch operations, multi‑table transactions, and non‑business data filtering.

BinlogCanalElasticsearch
0 likes · 13 min read
How to Build a Scalable Business Operation Log System with Canal and Elasticsearch
Top Architect
Top Architect
Nov 1, 2024 · Backend Development

Using Spring Transaction Hooks with TransactionSynchronizationManager for Asynchronous Kafka Messaging

The article explains how to leverage Spring's TransactionSynchronizationManager to detect active transactions and register synchronization callbacks that asynchronously send Kafka messages after a transaction commits, ensuring reliable fund‑flow archiving in payment systems while avoiding thread‑switch issues.

JavaKafkaSpring
0 likes · 11 min read
Using Spring Transaction Hooks with TransactionSynchronizationManager for Asynchronous Kafka Messaging
JD Retail Technology
JD Retail Technology
Oct 31, 2024 · Big Data

JDQ Kafka Bandwidth Throttling Architecture and Optimization

This article presents an in‑depth analysis of Kafka's native throttling mechanisms, identifies their limitations in large‑scale e‑commerce scenarios, and introduces JDQ's multi‑dimensional, dynamic throttling architecture that ensures stable throughput and priority‑aware bandwidth management across broker failures and traffic spikes.

JDQKafkabandwidth management
0 likes · 17 min read
JDQ Kafka Bandwidth Throttling Architecture and Optimization
Architect
Architect
Oct 30, 2024 · Backend Development

How to Build Distributed WebSocket Messaging with Spring, Redis, and Kafka

This article explains how to enable cross‑node WebSocket communication in a distributed Spring application by using a message queue (Redis or Kafka) to broadcast messages, tracking user connections with Redis, and providing a complete step‑by‑step implementation with code samples and configuration details.

JavaKafkaRedis
0 likes · 20 min read
How to Build Distributed WebSocket Messaging with Spring, Redis, and Kafka
Tencent Cloud Middleware
Tencent Cloud Middleware
Oct 30, 2024 · Backend Development

How Kafka Guarantees High Reliability and Performance – A Deep Technical Dive

This article thoroughly examines Apache Kafka’s architecture, covering its macro components, ack strategies, replication mechanisms, high‑watermark handling, leader election, and performance optimizations such as batch sending, compression, PageCache, zero‑copy, mmap and sendfile, while also explaining common pitfalls like data loss and log corruption.

KafkaPerformancedistributed systems
0 likes · 31 min read
How Kafka Guarantees High Reliability and Performance – A Deep Technical Dive
Rare Earth Juejin Tech Community
Rare Earth Juejin Tech Community
Oct 21, 2024 · Backend Development

Design and Implementation of a Live Streaming Danmaku System Using Redis, Kafka, and WebSocket

This article explains how to design a live‑streaming danmaku (bullet‑screen) feature by using Redis Zset as the underlying data structure, limiting the list to the latest ten messages, and handling retrieval via polling APIs or WebSocket, with Kafka for message buffering and a distributed lock to avoid ordering issues.

KafkaReal-time messagingRedis
0 likes · 8 min read
Design and Implementation of a Live Streaming Danmaku System Using Redis, Kafka, and WebSocket
ITPUB
ITPUB
Oct 19, 2024 · Backend Development

Why Some Teams Choose Redis Over Kafka for Message Queuing

The article examines the technical reasons a new tech director prefers using Redis Streams as a lightweight message queue instead of Kafka, comparing their features, operational costs, performance characteristics, and suitability for small‑to‑medium projects.

Backend DevelopmentKafka
0 likes · 7 min read
Why Some Teams Choose Redis Over Kafka for Message Queuing
Top Architect
Top Architect
Oct 18, 2024 · Backend Development

Using Spring Transaction Hooks to Send Kafka Messages After Commit

This article explains how to leverage Spring's TransactionSynchronizationManager to detect active transactions, register synchronization callbacks, and asynchronously send Kafka messages only after a transaction successfully commits, providing a practical example and detailed code snippets for building a reusable Spring Boot starter library.

JavaKafkaSpring
0 likes · 12 min read
Using Spring Transaction Hooks to Send Kafka Messages After Commit
Architect
Architect
Oct 16, 2024 · Backend Development

Mastering Spring Transaction Hooks for Safe Kafka Publishing

This article explains how to use Spring's TransactionSynchronizationManager to detect active transactions, register synchronization callbacks, and asynchronously publish payment‑log messages to Kafka without affecting the main business flow, complete with practical code snippets and pitfalls to avoid.

BackendJavaKafka
0 likes · 11 min read
Mastering Spring Transaction Hooks for Safe Kafka Publishing
Sanyou's Java Diary
Sanyou's Java Diary
Oct 10, 2024 · Big Data

Why Kafka Handles Millions of Messages Per Second: Inside Its High‑Performance Architecture

This article explains how Kafka achieves ultra‑high throughput and low latency despite being disk‑based, covering its Reactor I/O network model, zero‑copy techniques, partitioning strategies, segment logs with sparse indexes, sequential disk writes, page cache usage, compression, batch processing, and lock‑free offset management.

KafkaPerformanceReactor I/O
0 likes · 27 min read
Why Kafka Handles Millions of Messages Per Second: Inside Its High‑Performance Architecture
Sanyou's Java Diary
Sanyou's Java Diary
Sep 19, 2024 · Operations

Message Queue Showdown: Choosing Kafka, Pulsar, RabbitMQ, RocketMQ or NSQ

This comprehensive guide compares five popular message queue systems—Kafka, Pulsar, RabbitMQ, RocketMQ, and NSQ—covering their architectures, core concepts, selection criteria, feature differences such as ordering, latency, durability, scaling, and operational considerations, to help engineers select the most suitable solution for their use cases.

ArchitectureKafkaNSQ
0 likes · 46 min read
Message Queue Showdown: Choosing Kafka, Pulsar, RabbitMQ, RocketMQ or NSQ
Su San Talks Tech
Su San Talks Tech
Sep 18, 2024 · Backend Development

Unlocking Kafka’s Million‑Message‑Per‑Second Performance: Key Techniques Explained

This article explains how Kafka attains extremely high throughput by using batch sending, message compression, sequential disk I/O, PageCache, zero‑copy transfers, and memory‑mapped index files, detailing the underlying code paths and trade‑offs that enable millions of messages per second.

Batch ProcessingCompressionHigh Performance
0 likes · 12 min read
Unlocking Kafka’s Million‑Message‑Per‑Second Performance: Key Techniques Explained
Code Ape Tech Column
Code Ape Tech Column
Sep 12, 2024 · Backend Development

Distributed WebSocket Messaging with Redis and Kafka in Spring

This article explains how to enable cross‑server WebSocket communication in a distributed Spring application by using a message queue such as Redis or Kafka, detailing two architectural approaches, concrete code examples, configuration files, and a sample client page.

BackendKafkaRedis
0 likes · 17 min read
Distributed WebSocket Messaging with Redis and Kafka in Spring
MaGe Linux Operations
MaGe Linux Operations
Sep 10, 2024 · Backend Development

How Kafka Elects Leaders and Distributes Partitions: A Deep Dive

This article explains Kafka's leader election process, partition assignment strategy, distribution policies, file layout, and the evolution of consumer offset storage, providing a comprehensive overview of how Kafka ensures reliable and efficient message handling in a distributed environment.

KafkaLeader ElectionPartition Assignment
0 likes · 5 min read
How Kafka Elects Leaders and Distributes Partitions: A Deep Dive
Tencent Cloud Developer
Tencent Cloud Developer
Sep 6, 2024 · Backend Development

Message Queue (MQ) Overview, Selection Criteria, and Comparison of Kafka, Pulsar, RocketMQ, RabbitMQ, and NSQ

The article surveys major message‑queue systems—Kafka, Pulsar, RocketMQ, RabbitMQ, and NSQ—detailing their architectures, key features, selection criteria such as ordering, scalability and latency, performance benchmarks, operational considerations, and typical use cases, and advises which platform best fits high‑throughput, reliability, low‑latency, or lightweight Go‑centric scenarios.

KafkaNSQPulsar
0 likes · 47 min read
Message Queue (MQ) Overview, Selection Criteria, and Comparison of Kafka, Pulsar, RocketMQ, RabbitMQ, and NSQ
Top Architect
Top Architect
Sep 3, 2024 · Backend Development

Design and Implementation of a General Asynchronous Processing SDK for Java Backend

This article introduces a reusable asynchronous processing SDK for Java backend systems, explaining its purpose, advantages, underlying principles, component choices, design patterns, database schema, configuration options, usage instructions, and best‑practice notes, while providing complete code snippets and deployment details.

@AsyncBackendDesign Pattern
0 likes · 13 min read
Design and Implementation of a General Asynchronous Processing SDK for Java Backend
JavaEdge
JavaEdge
Sep 1, 2024 · Backend Development

How Kafka Implements Transactions and Achieves Exactly‑Once Guarantees

This article explains how Kafka implements transactions using a two‑phase commit protocol, compares its approach with RocketMQ, details the exactly‑once semantics for streaming workloads, and walks through the coordinator roles, message flow, and practical use‑case scenarios.

Kafkatransactiontwo-phase commit
0 likes · 10 min read
How Kafka Implements Transactions and Achieves Exactly‑Once Guarantees
21CTO
21CTO
Aug 28, 2024 · Backend Development

Why Message Queues Are Essential for Scalable Backend Systems

This article explains how message queues decouple and buffer tasks such as payment processing, inventory updates, and notifications in high‑traffic online stores, improving reliability, scalability, and fault tolerance through asynchronous communication.

KafkaRabbitMQasynchronous-processing
0 likes · 6 min read
Why Message Queues Are Essential for Scalable Backend Systems
Su San Talks Tech
Su San Talks Tech
Aug 26, 2024 · Backend Development

7 Real-World Message Queue Patterns Every Backend Engineer Should Know

This article shares seven classic message‑queue use cases—from asynchronous decoupling and peak‑smoothing to delayed tasks, broadcast consumption, distributed transactions, and data hub integration—illustrated with real‑world examples, diagrams, and RocketMQ code snippets.

KafkaRocketMQ
0 likes · 12 min read
7 Real-World Message Queue Patterns Every Backend Engineer Should Know
Architect's Journey
Architect's Journey
Aug 19, 2024 · Backend Development

Flexible Multi-MQ Component Update: Switch Between RocketMQ, Redis, Kafka, and RabbitMQ

The article explains why adopting a message‑queue improves microservice decoupling and responsibility boundaries, then details the design of a pluggable Base‑MQ component—including @MQEventListener, MQEvent base class, configuration, and MQClient interface—followed by concrete implementations for RocketMQ, Kafka, Redis and RabbitMQ, an example event flow, and guidance for extending the framework.

JavaKafkaMicroservices
0 likes · 27 min read
Flexible Multi-MQ Component Update: Switch Between RocketMQ, Redis, Kafka, and RabbitMQ
Hacker Afternoon Tea
Hacker Afternoon Tea
Aug 16, 2024 · Backend Development

Building a Fast Real‑Time Streaming Platform with Kafka and ClickHouse

This guide walks through the challenges of ingesting high‑volume sales‑point data into ClickHouse, explains why a buffering layer like Kafka is essential, compares three integration approaches—ClickHouse Kafka engine, Kafka Connect, and an external writer—and provides concrete SQL and Terraform examples to help you choose the right solution.

ClickHouseExternal writerKafka
0 likes · 13 min read
Building a Fast Real‑Time Streaming Platform with Kafka and ClickHouse
Efficient Ops
Efficient Ops
Aug 14, 2024 · Operations

Building a Real-Time Log Monitoring System with ELK, Kafka, and Python

This article details how to construct a log‑monitoring platform using the ELK stack, Kafka buffering, and a Python scheduler to collect, process, and alert on error logs, offering practical configuration tips and performance optimizations for production environments.

ELKElasticsearchKafka
0 likes · 10 min read
Building a Real-Time Log Monitoring System with ELK, Kafka, and Python
StarRocks
StarRocks
Aug 14, 2024 · Big Data

Mastering StarRocks & Apache Paimon: A Fast‑Track Lakehouse Guide

This guide provides a comprehensive overview of Apache Paimon’s architecture, key features, and advantages, explains how to integrate it with StarRocks for real‑time lakehouse analytics, and walks through a complete quick‑start setup including component installation, Flink and Kafka deployment, data ingestion, table creation, and query execution with time‑travel support.

Apache PaimonFlinkKafka
0 likes · 18 min read
Mastering StarRocks & Apache Paimon: A Fast‑Track Lakehouse Guide
Java Interview Crash Guide
Java Interview Crash Guide
Aug 13, 2024 · Backend Development

How to Build a Robust Asynchronous Processing SDK with Spring and Kafka

This article explains the design, advantages, and implementation details of a generic asynchronous processing SDK that leverages Spring transactional events, Kafka messaging, XXL‑Job scheduling, and a non‑intrusive architecture to ensure data consistency, fault tolerance, and easy integration across services.

Design PatternsKafkaSpring
0 likes · 11 min read
How to Build a Robust Asynchronous Processing SDK with Spring and Kafka
Top Architect
Top Architect
Aug 7, 2024 · Backend Development

Kafka Consumer Deep Dive: Offset Management, Rebalance Strategies, and Thread Safety

This article explains Kafka consumer semantics such as at‑most‑once, at‑least‑once and exactly‑once delivery, shows how to configure and commit offsets, describes the rebalance process and partition‑assignment strategies, and discusses thread‑safety and task scheduling with illustrative Java code examples.

ConsumerKafkaRebalance
0 likes · 17 min read
Kafka Consumer Deep Dive: Offset Management, Rebalance Strategies, and Thread Safety
Java Backend Technology
Java Backend Technology
Aug 6, 2024 · Backend Development

Master Spring Boot Project Setup: From IDE to Version Management & Common Tools

This guide walks through initializing a Spring Boot project, handling IDE licensing hurdles, ensuring compatible Spring Cloud, Spring Boot, and Kafka versions, leveraging Maven for dependency management, and integrating essential utilities like global exception handling, logging, CORS, Swagger, and helpful development tools.

BackendDevOpsKafka
0 likes · 11 min read
Master Spring Boot Project Setup: From IDE to Version Management & Common Tools
Code Ape Tech Column
Code Ape Tech Column
Aug 5, 2024 · Backend Development

Design and Implementation of a Generic Asynchronous Processing SDK for Java Backend

This article describes a Java‑based asynchronous processing SDK that leverages Spring AOP, transactional event listeners, Kafka, XXL‑Job, and MySQL to provide non‑intrusive, reliable, and eventually consistent background execution of business methods while preserving transaction integrity and offering configurable retry and compensation strategies.

AOPAsyncJava
0 likes · 8 min read
Design and Implementation of a Generic Asynchronous Processing SDK for Java Backend
Architect
Architect
Aug 3, 2024 · Big Data

Kafka Storage Architecture Design: Deep Analysis and Implementation

This article thoroughly examines Kafka's storage architecture, analyzing its design motivations, storage mechanisms, log formats, partitioning, indexing, cleanup strategies, and performance optimizations such as sequential log writes, sparse indexing, page cache, and zero‑copy, providing insights applicable to other storage systems.

KafkaLog SegmentsSparse Index
0 likes · 22 min read
Kafka Storage Architecture Design: Deep Analysis and Implementation
macrozheng
macrozheng
Aug 2, 2024 · Backend Development

How to Quickly Resolve Massive Kafka Message Backlog in Production

This guide explains why Kafka message backlogs occur, how to diagnose bugs, optimize consumer logic, and use temporary topics for emergency scaling, while emphasizing monitoring, alerts, and proper offset handling to keep your streaming system healthy.

BacklogConsumerJava
0 likes · 5 min read
How to Quickly Resolve Massive Kafka Message Backlog in Production
Top Architect
Top Architect
Jul 31, 2024 · Backend Development

Design and Implementation of a Generic Asynchronous Processing SDK for Spring Applications

This article introduces a Spring‑based asynchronous processing SDK that leverages annotations, transaction event listeners, Kafka, XXL‑Job, and a dedicated database to achieve non‑blocking execution, fault tolerance, and eventual consistency while preserving transaction integrity and providing a clear configuration and usage guide.

BackendDesign PatternKafka
0 likes · 12 min read
Design and Implementation of a Generic Asynchronous Processing SDK for Spring Applications
Programmer XiaoFu
Programmer XiaoFu
Jul 24, 2024 · Backend Development

Root‑Cause Solutions for Kafka Message Duplicates

The article analyzes why Kafka messages are duplicated on both producer and consumer sides, explains at‑most‑once, at‑least‑once, and exactly‑once delivery semantics, and walks through three concrete remedies—idempotent producers, transactions, and consumer‑side deduplication—backed by configuration details, code samples, and practical deployment steps.

ConfigurationConsumer IdempotencyKafka
0 likes · 12 min read
Root‑Cause Solutions for Kafka Message Duplicates
Soul Technical Team
Soul Technical Team
Jul 23, 2024 · Big Data

Kafka Stability Challenges and Governance Framework at Soul

This article analyzes the role, application scenarios, stability challenges, and comprehensive governance framework of Apache Kafka at Soul, covering deployment, configuration, monitoring, standard controls, common misuse, and future directions toward cloud‑native solutions.

KafkaOperationsStreaming
0 likes · 30 min read
Kafka Stability Challenges and Governance Framework at Soul
Rare Earth Juejin Tech Community
Rare Earth Juejin Tech Community
Jul 22, 2024 · Big Data

Comprehensive Guide to Kafka: Architecture, Core Concepts, and Configuration

This article provides an in‑depth overview of Apache Kafka, covering its use cases, comparison with other message queues, versioning, performance mechanisms, core concepts such as topics, partitions, offsets, consumer groups, rebalancing, replication, leader election, idempotence, transactions, compression, interceptors, request handling, and practical configuration tips for reliable streaming applications.

ConsumerKafkaStreaming
0 likes · 25 min read
Comprehensive Guide to Kafka: Architecture, Core Concepts, and Configuration
Architect
Architect
Jul 12, 2024 · Backend Development

How to Prevent Message Loss and Duplication in Kafka and RocketMQ

This article analyzes the root causes of message loss and duplication in modern message queues, explains how synchronous and asynchronous sending affect reliability, compares storage strategies like sync‑flush and broker clustering, and offers concrete idempotent handling techniques using database constraints and Redis.

BackendKafkaMessage reliability
0 likes · 8 min read
How to Prevent Message Loss and Duplication in Kafka and RocketMQ
Architect's Guide
Architect's Guide
Jul 11, 2024 · Backend Development

Using Spring Transaction Hooks to Send Kafka Messages After Commit

This article demonstrates how to leverage Spring's TransactionSynchronizationManager to detect active transactions and register synchronization callbacks that asynchronously push payment‑ledger messages to Kafka only after the transaction successfully commits, ensuring data consistency and minimal impact on the main business flow.

BackendJavaKafka
0 likes · 9 min read
Using Spring Transaction Hooks to Send Kafka Messages After Commit
Mike Chen's Internet Architecture
Mike Chen's Internet Architecture
Jul 3, 2024 · Backend Development

Preventing Duplicate Message Consumption in Kafka

This article explains why duplicate message consumption occurs in Kafka, outlines the underlying confirmation and consumer failure issues, and presents idempotent design strategies such as deduplication tables and code examples to ensure reliable processing.

Backend DevelopmentKafkaduplicate-consumption
0 likes · 5 min read
Preventing Duplicate Message Consumption in Kafka
360 Smart Cloud
360 Smart Cloud
Jul 3, 2024 · Operations

Practical Practices for Enhancing Kafka Cluster Stability at 360

This article details 360's comprehensive approach to improving Apache Kafka cluster stability through proactive operations, capacity assessment, parameter tuning, monitoring, version upgrades, and traffic control, offering concrete guidelines and best‑practice recommendations for large‑scale message‑queue deployments.

ClusterKafkamonitoring
0 likes · 33 min read
Practical Practices for Enhancing Kafka Cluster Stability at 360
Java Architect Essentials
Java Architect Essentials
Jun 26, 2024 · Databases

Why Organizations Should Consider Using Apache Kafka Instead of Relational Databases

This article explains why organizations may replace traditional relational databases with Apache Kafka as a system of record, highlighting Kafka's economic, scalable, immutable log capabilities, event replay, flexibility for diverse use cases, and its suitability for highly regulated, data‑intensive environments.

Data StreamingImmutable LogKafka
0 likes · 10 min read
Why Organizations Should Consider Using Apache Kafka Instead of Relational Databases
Architect
Architect
Jun 25, 2024 · Backend Development

Mastering Spring Transaction Hooks: Send Kafka Messages Only After Commit

This article explains how to use Spring's TransactionSynchronizationManager to detect active transactions and register callbacks that asynchronously send Kafka messages after a transaction successfully commits, ensuring data consistency in payment‑system archiving scenarios.

BackendJavaKafka
0 likes · 10 min read
Mastering Spring Transaction Hooks: Send Kafka Messages Only After Commit
macrozheng
macrozheng
Jun 25, 2024 · Backend Development

7 Real-World Message Queue Patterns Every Backend Engineer Should Master

This article explores seven classic message‑queue use cases—from asynchronous decoupling and traffic‑shaping to delayed tasks, broadcast consumption, distributed transactions, and using Kafka as a data hub—illustrated with real‑world examples and code snippets.

Broadcast ConsumptionDelayed MessageKafka
0 likes · 12 min read
7 Real-World Message Queue Patterns Every Backend Engineer Should Master
Wukong Talks Architecture
Wukong Talks Architecture
Jun 20, 2024 · Backend Development

Seven Classic Use Cases of Message Queues

This article shares seven practical scenarios—such as asynchronous processing, traffic smoothing, message bus, delayed tasks, broadcast consumption, distributed transactions, and data hub integration—where message queues like RocketMQ, Kafka, ActiveMQ, and RabbitMQ help solve high‑concurrency challenges in modern backend systems.

Broadcast ConsumptionDelay MessagesKafka
0 likes · 11 min read
Seven Classic Use Cases of Message Queues
Su San Talks Tech
Su San Talks Tech
Jun 2, 2024 · Big Data

Mastering Kafka: Core Architecture, Use Cases, and Design Principles

This article provides a comprehensive overview of Apache Kafka, covering its role as a message queue, core components, topic and partition design, consumer groups, storage mechanisms, high‑availability features, delivery guarantees, ZooKeeper coordination, and scalability strategies for building robust real‑time data pipelines.

KafkaStreamingbig data
0 likes · 15 min read
Mastering Kafka: Core Architecture, Use Cases, and Design Principles
Xiaohongshu Tech REDtech
Xiaohongshu Tech REDtech
May 23, 2024 · Cloud Native

Cloud-Native Architecture and Tiered Storage for Xiaohongshu Kafka: Cost Reduction, Elastic Migration, and Performance Optimization

Xiaohongshu's big-data storage team built cloud-native architecture with tiered storage, containerized Kafka, and custom load balancer, cutting storage costs up to 60%, enabling minute‑level elastic migration, improving scaling efficiency tenfold, and boosting performance via caching and batch reads.

Elastic ScalingKafkacost optimization
0 likes · 20 min read
Cloud-Native Architecture and Tiered Storage for Xiaohongshu Kafka: Cost Reduction, Elastic Migration, and Performance Optimization
Qunar Tech Salon
Qunar Tech Salon
May 20, 2024 · Big Data

Optimizing Kafka Production at Qunar Travel: Reducing CPU Usage by 2000 Cores

This article presents a comprehensive case study of how Qunar Travel identified and resolved Kafka production bottlenecks—through metric monitoring, thread and flush parameter tuning, and Filebeat batch adjustments—resulting in a 2000‑core CPU reduction, higher network idle rates, and lower resource consumption across three clusters.

Kafkamonitoring
0 likes · 12 min read
Optimizing Kafka Production at Qunar Travel: Reducing CPU Usage by 2000 Cores
Su San Talks Tech
Su San Talks Tech
May 14, 2024 · Backend Development

How We Fixed Repeated Kafka Message Backlog in a High‑Traffic Restaurant System

This article details a series of Kafka message backlog incidents in a busy restaurant ordering system, analyzes root causes such as consumer speed, database indexing, and sudden traffic spikes, and presents step‑by‑step optimizations—including batch queries, index tuning, data archiving, and thread‑pool adjustments—to restore real‑time performance.

KafkaMySQLPerformance Optimization
0 likes · 11 min read
How We Fixed Repeated Kafka Message Backlog in a High‑Traffic Restaurant System
Mike Chen's Internet Architecture
Mike Chen's Internet Architecture
May 11, 2024 · Big Data

Comprehensive Introduction to Apache Kafka: Architecture, Features, and Use Cases

This article provides a detailed overview of Apache Kafka, covering its core characteristics, distributed architecture, key components such as topics, partitions, brokers, producers, consumers, ZooKeeper, and common application scenarios like log collection, event‑driven architecture, real‑time analytics, and monitoring.

ArchitectureKafkaStreaming
0 likes · 7 min read
Comprehensive Introduction to Apache Kafka: Architecture, Features, and Use Cases
Lobster Programming
Lobster Programming
May 9, 2024 · Backend Development

Detect Hot Items in High‑Concurrency Sales Using OpenResty, Kafka & Elasticsearch

During flash‑sale events like JD or Taobao, a tiny fraction of products generate the majority of traffic, causing server overload; this guide explains how to identify and collect hotspot product data in real time using an OpenResty‑Lua front‑end, Kafka for messaging, and Elasticsearch for storage and analysis.

ElasticsearchHot ItemsKafka
0 likes · 5 min read
Detect Hot Items in High‑Concurrency Sales Using OpenResty, Kafka & Elasticsearch