Skip to content
Tagged articles

Kafka

500 articles · Page 4 of 5
Su San Talks Tech
Su San Talks Tech
May 23, 2025 · Backend Development

Kafka vs RabbitMQ vs RocketMQ vs ActiveMQ: Key Differences and How to Choose

This article explains the fundamentals, architectures, key concepts, advantages, and disadvantages of Kafka, RabbitMQ, RocketMQ, and ActiveMQ, and provides practical guidance on selecting the most suitable message queue for various application scenarios such as high‑throughput logging, financial transactions, or small‑scale services.

KafkaRabbitMQRocketMQ
0 likes · 19 min read
Kafka vs RabbitMQ vs RocketMQ vs ActiveMQ: Key Differences and How to Choose
Mike Chen's Internet Architecture
Mike Chen's Internet Architecture
May 22, 2025 · Big Data

Optimizing Kafka Producer for High Throughput: Batching, Asynchronous Sending, Compression, and Concurrency

This article details practical techniques for boosting Kafka producer throughput, covering batch sending optimization, linger.ms tuning, asynchronous send handling, compression options, and concurrent sending strategies, while providing code examples and performance considerations for high‑throughput streaming applications.

Kafkaasynchronousbatching
0 likes · 4 min read
Optimizing Kafka Producer for High Throughput: Batching, Asynchronous Sending, Compression, and Concurrency
Full-Stack Internet Architecture
Full-Stack Internet Architecture
May 20, 2025 · Big Data

Why Learn Kafka? Core Benefits, Use Cases, and a Summary

This article explains why Kafka is widely adopted by top companies, outlines its high throughput, scalability, and durability, and describes key real‑time data pipeline, stream processing, and big‑data integration scenarios, concluding that mastering Kafka is essential for modern backend and data engineering roles.

Data EngineeringKafkareal-time processing
0 likes · 4 min read
Why Learn Kafka? Core Benefits, Use Cases, and a Summary
Code Ape Tech Column
Code Ape Tech Column
May 19, 2025 · Backend Development

Mastering CQRS in Spring Microservices: From Basics to Event Sourcing

This article explains the CQRS architectural pattern, its benefits and pitfalls, and provides a step‑by‑step guide to implementing CQRS with Spring Boot, Axon, and Kafka, including command and query handling, event sourcing, and practical considerations for microservice architectures.

AxonBackend ArchitectureCQRS
0 likes · 13 min read
Mastering CQRS in Spring Microservices: From Basics to Event Sourcing
Selected Java Interview Questions
Selected Java Interview Questions
May 15, 2025 · Backend Development

Six Common Approaches to Synchronize MySQL Data to Elasticsearch

This article reviews six mainstream solutions for keeping MySQL and Elasticsearch in sync—including synchronous double‑write, asynchronous MQ‑based double‑write, Logstash polling, Canal binlog listening, DataX batch migration, and Flink stream processing—detailing their scenarios, advantages, drawbacks, and practical code examples to guide optimal technical selection.

CanalElasticsearchFlink
0 likes · 8 min read
Six Common Approaches to Synchronize MySQL Data to Elasticsearch
Airbnb Technology Team
Airbnb Technology Team
May 13, 2025 · Backend Development

How Airbnb Scaled Its Postcard Generation System for Group Travel

Airbnb's media team engineered a scalable postcard generation pipeline that matches destinations, handles localized text layout, and delivers high‑performance images across iOS, Android, Web, and external messaging platforms using flexible templates, async processing, and pre‑generation strategies.

AirbnbAsynchronous ProcessingBackend Architecture
0 likes · 11 min read
How Airbnb Scaled Its Postcard Generation System for Group Travel
Code Ape Tech Column
Code Ape Tech Column
May 9, 2025 · Databases

Efficient Strategies for Importing One Billion Records into MySQL

This article explains how to import 1 billion 1 KB log records stored in HDFS or S3 into MySQL by analyzing single‑table limits, using batch inserts, choosing storage engines, sharding, optimizing file‑reading methods, and coordinating distributed tasks with Redis, Redisson, and Zookeeper to ensure ordered, reliable, and high‑throughput data loading.

KafkaMySQLPerformance Tuning
0 likes · 19 min read
Efficient Strategies for Importing One Billion Records into MySQL
Raymond Ops
Raymond Ops
May 7, 2025 · Operations

How Kafka Elects Leaders and Distributes Partitions: Inside the Mechanics

Kafka’s internal mechanisms for leader election, partition assignment, and file storage are explained, covering how the Controller uses ZooKeeper, the ISR-based leader selection process, partition distribution strategies, segment file structures, and the evolution of offset management from Zookeeper to the __consumer_offsets topic.

KafkaOffsetsPartition Assignment
0 likes · 5 min read
How Kafka Elects Leaders and Distributes Partitions: Inside the Mechanics
Su San Talks Tech
Su San Talks Tech
May 5, 2025 · Big Data

6 Proven Ways to Sync MySQL Data to Elasticsearch – Choose the Right Strategy

This article compares six mainstream MySQL‑to‑Elasticsearch synchronization methods—synchronous double‑write, asynchronous MQ, Logstash polling, Canal binlog listening, DataX batch sync, and Flink streaming—detailing scenarios, code samples, advantages, drawbacks, and practical selection guidance for developers.

CanalElasticsearchFlink
0 likes · 9 min read
6 Proven Ways to Sync MySQL Data to Elasticsearch – Choose the Right Strategy
dbaplus Community
dbaplus Community
Apr 16, 2025 · Backend Development

How Ctrip’s Kafka Gatekeeper Boosts FinOps Data Quality and Automates Cost Governance

This article explains how Ctrip’s hybrid‑cloud FinOps billing system uses a custom Kafka Gatekeeper to detect, locate, and automatically remediate data‑quality issues across dozens of self‑built PaaS services, improving coverage, timeliness, and responsibility attribution while supporting high‑availability deployments.

BackendCloud NativeFinOps
0 likes · 19 min read
How Ctrip’s Kafka Gatekeeper Boosts FinOps Data Quality and Automates Cost Governance
Java Captain
Java Captain
Apr 14, 2025 · Backend Development

Project Initialization, Version Management, and Scaffold Setup for Spring Boot Backend Development

This guide walks through initializing a Spring Boot project, handling IDE setup challenges, managing version compatibility between Spring Cloud, Spring Boot, and Kafka, and provides practical code examples for global exception handling, logging, CORS configuration, Swagger integration, and useful development tools.

Exception HandlingKafkaMaven
0 likes · 12 min read
Project Initialization, Version Management, and Scaffold Setup for Spring Boot Backend Development
macrozheng
macrozheng
Apr 11, 2025 · Backend Development

How to Guarantee Exactly‑Once Message Consumption in High‑Concurrency Systems

This article explains common causes of duplicate message consumption in high‑traffic systems and presents a three‑layer defense—producer idempotence, broker de‑duplication, and consumer idempotent design—plus monitoring and reconciliation strategies to achieve reliable exactly‑once processing.

KafkaMessage QueueRocketMQ
0 likes · 7 min read
How to Guarantee Exactly‑Once Message Consumption in High‑Concurrency Systems
Sanyou's Java Diary
Sanyou's Java Diary
Apr 10, 2025 · Backend Development

Why RocketMQ Beats Kafka: Architecture Simplified and Features Amplified

This article explains how RocketMQ, a Chinese‑origin message queue, simplifies Kafka’s architecture while adding powerful features such as tag‑based filtering, transactional messaging, delayed and dead‑letter queues, and a unified commit‑log storage model, making delayed processing and high‑throughput scenarios easier to implement.

KafkaMessage QueueRocketMQ
0 likes · 10 min read
Why RocketMQ Beats Kafka: Architecture Simplified and Features Amplified
Architecture Digest
Architecture Digest
Apr 6, 2025 · Backend Development

Design and Implementation of a General‑Purpose Asynchronous Processing SDK for Backend Systems

This article introduces a reusable asynchronous processing SDK built on Spring, Kafka, and MySQL that leverages @AsyncExec annotations, transactional event listeners, and configurable thread pools to ensure reliable, non‑blocking execution, data consistency, and fault‑tolerant handling of business logic in backend applications.

BackendKafkaasync
0 likes · 8 min read
Design and Implementation of a General‑Purpose Asynchronous Processing SDK for Backend Systems
Ma Wei Says
Ma Wei Says
Mar 30, 2025 · Fundamentals

How Kafka 4.0’s KRaft Replaces ZooKeeper with Raft Consensus

Kafka 4.0 introduces KRaft, a ZooKeeper‑free metadata layer built on the Raft consensus algorithm, detailing role transitions, leader election, log replication, controller and broker responsibilities, and fault‑tolerance mechanisms, enabling a more scalable and self‑managed architecture for large‑scale distributed streaming.

Consensus AlgorithmKRaftKafka
0 likes · 13 min read
How Kafka 4.0’s KRaft Replaces ZooKeeper with Raft Consensus
Ma Wei Says
Ma Wei Says
Mar 28, 2025 · Backend Development

Choosing the Right Message Queue: Kafka vs RocketMQ vs RabbitMQ Explained

This article compares Kafka, RocketMQ, and RabbitMQ, detailing their architectures, performance characteristics, strengths, and ideal use‑cases to help engineers select the most suitable message‑queue solution for high‑throughput, fault‑tolerant, and real‑time processing scenarios.

KafkaRabbitMQRocketMQ
0 likes · 11 min read
Choosing the Right Message Queue: Kafka vs RocketMQ vs RabbitMQ Explained
Linux Cloud Computing Practice
Linux Cloud Computing Practice
Mar 28, 2025 · Backend Development

200+ Essential Interview Questions for Linux, Nginx, MySQL, Kafka & Security

This article compiles a comprehensive collection of over two hundred interview questions covering Linux system administration, Nginx web server, MySQL databases, Redis caching, Kafka messaging, and core network security topics, providing a one‑stop resource for candidates preparing for major tech company interviews.

Interview QuestionsKafkaMySQL
0 likes · 11 min read
200+ Essential Interview Questions for Linux, Nginx, MySQL, Kafka & Security
Didi Tech
Didi Tech
Mar 27, 2025 · Operations

Performance Optimization and Architecture of iLogTail for High‑Scale Log Collection

Didi replaced its legacy agent with Alibaba’s open‑source iLogTail, re‑architected it to use a shared thread‑pool and SIMD‑accelerated parsing, rewrote critical plugins in C++ and added robust Kafka retry logic, achieving over twice the throughput while cutting CPU usage by more than half and maintaining near‑zero latency at massive scale.

C++KafkaLog Collection
0 likes · 10 min read
Performance Optimization and Architecture of iLogTail for High‑Scale Log Collection
Linux Ops Smart Journey
Linux Ops Smart Journey
Mar 27, 2025 · Cloud Native

Stream Kubernetes Events to Elasticsearch with Kafka & Logstash

This guide walks you through deploying the Kubernetes Event Exporter, packaging its Helm chart and Docker image, configuring Kafka TLS secrets, setting up Logstash to ingest events and forward them to Elasticsearch, creating an index template, and verifying the end‑to‑end pipeline.

ElasticsearchEvent ExporterHelm
0 likes · 10 min read
Stream Kubernetes Events to Elasticsearch with Kafka & Logstash
iQIYI Technical Product Team
iQIYI Technical Product Team
Mar 27, 2025 · Big Data

Cost‑Effective Real‑Time Data Warehouse 2.0: Migrating from Kafka to Iceberg

iQIYI transformed its real‑time data warehouse by replacing a costly Kafka‑based Lambda stack with a unified stream‑batch Iceberg lake, cutting storage expenses by 90%, halving compute costs, extending data retention, and delivering minute‑level freshness for 90% of use cases while preserving second‑level processing where needed.

FlinkIcebergKafka
0 likes · 11 min read
Cost‑Effective Real‑Time Data Warehouse 2.0: Migrating from Kafka to Iceberg
Su San Talks Tech
Su San Talks Tech
Mar 27, 2025 · Operations

How to Ensure Data Consistency in Message Queues: 10 Hard‑Earned Lessons

This article explores why message queues can lose consistency, presents concrete solutions such as transactional two‑phase commits, persistence settings, replica configurations, unique IDs, idempotent designs, and dead‑letter queues, and shares ten practical lessons drawn from real‑world incidents.

KafkaRabbitMQTransactional Messaging
0 likes · 12 min read
How to Ensure Data Consistency in Message Queues: 10 Hard‑Earned Lessons
Big Data Technology Architecture
Big Data Technology Architecture
Mar 25, 2025 · Big Data

Kafka 4.0 Release: KRaft Architecture, Consumer Group Optimizations, and New Queue Features

Kafka 4.0 marks a milestone release that replaces ZooKeeper with the KRaft consensus engine, improves scalability and performance, introduces a server‑side consumer‑group protocol, adds shared‑group queue capabilities, and updates Java requirements and documentation, delivering a more robust and flexible streaming platform.

Java11KRaftKafka
0 likes · 6 min read
Kafka 4.0 Release: KRaft Architecture, Consumer Group Optimizations, and New Queue Features
Big Data Technology & Architecture
Big Data Technology & Architecture
Mar 24, 2025 · Big Data

Apache Kafka 4.0: Major New Features – KRaft Architecture, Consumer Group Protocol, Queue Mode, Java Upgrade, API Simplifications and More

Apache Kafka 4.0 introduces a groundbreaking KRaft architecture that removes ZooKeeper, a revamped consumer‑group protocol that dramatically speeds up rebalancing, a new queue mode for point‑to‑point messaging, upgraded Java requirements, streamlined APIs, and numerous performance and security enhancements, reshaping both development and operations for large‑scale streaming workloads.

Java 11Java 17KRaft
0 likes · 12 min read
Apache Kafka 4.0: Major New Features – KRaft Architecture, Consumer Group Protocol, Queue Mode, Java Upgrade, API Simplifications and More
Code Ape Tech Column
Code Ape Tech Column
Mar 14, 2025 · Backend Development

Eight Common Use Cases of Message Queues in Backend Development

This article explores eight common scenarios for using message queues in backend development, covering asynchronous processing, service decoupling, traffic shaping, delayed tasks, log aggregation, distributed transactions, remote calls, and broadcast notifications, each illustrated with Java, RocketMQ, and Kafka code examples.

Asynchronous ProcessingJavaKafka
0 likes · 15 min read
Eight Common Use Cases of Message Queues in Backend Development
Java Tech Enthusiast
Java Tech Enthusiast
Mar 9, 2025 · Backend Development

Handling Duplicate Messages in Message Queues

Message queues can deliver duplicates under at‑least‑once semantics, so to protect idempotent business logic such as orders and payments you should combine producer‑side idempotence (e.g., Kafka’s enable.idempotence), broker‑side deduplication (e.g., Pulsar), and a consumer‑side guard using unique IDs stored in a database or Redis.

KafkaMessage QueuePulsar
0 likes · 7 min read
Handling Duplicate Messages in Message Queues
IT Services Circle
IT Services Circle
Mar 8, 2025 · Backend Development

Handling Duplicate Messages in Message Queues: Semantics, Producer and Broker Deduplication, and Consumer Strategies

Message queues can cause duplicate messages that affect idempotent business processes, so this article explains the three delivery semantics (At Least Once, Exactly Once, At Most Once), the causes of duplication, and practical deduplication techniques for producers, brokers (Kafka, Pulsar), and consumers using code examples.

DeduplicationKafkaPulsar
0 likes · 8 min read
Handling Duplicate Messages in Message Queues: Semantics, Producer and Broker Deduplication, and Consumer Strategies
Selected Java Interview Questions
Selected Java Interview Questions
Mar 5, 2025 · Backend Development

Spring Boot Project Initialization, Version Management, and Common Backend Tools

This article walks through setting up a Spring Boot project, handling version compatibility with Spring Cloud, Kafka, and Maven, demonstrates global exception handling, logging, CORS and Swagger configurations, and recommends useful backend utilities such as embedded Redis, MyBatis Plus, and Redisson.

Backend DevelopmentCORSException Handling
0 likes · 9 min read
Spring Boot Project Initialization, Version Management, and Common Backend Tools
Linux Ops Smart Journey
Linux Ops Smart Journey
Feb 27, 2025 · Information Security

Build Military‑Grade Kafka Security in Four Easy Steps

This guide walks you through four progressive stages—basic configuration, SSL encryption, SCRAM authentication, and combined SSL+SASL—showing how to harden Kafka with certificates, keystores, and client settings to achieve financial‑level protection.

KafkaSASLSCRAM
0 likes · 9 min read
Build Military‑Grade Kafka Security in Four Easy Steps
dbaplus Community
dbaplus Community
Feb 26, 2025 · Operations

Choosing the Right Message Queue: A Deep Dive into Kafka, Pulsar, RocketMQ, RabbitMQ, and NSQ

This comprehensive guide explains the role of message queues in distributed systems, compares the architectures and core concepts of Kafka, Pulsar, RocketMQ, RabbitMQ, and NSQ, and provides practical criteria for selecting, configuring, and operating a message‑queue solution in high‑performance, high‑availability environments.

KafkaNSQPulsar
0 likes · 50 min read
Choosing the Right Message Queue: A Deep Dive into Kafka, Pulsar, RocketMQ, RabbitMQ, and NSQ
macrozheng
macrozheng
Feb 20, 2025 · Backend Development

Master Java Concurrency & SpringBoot: DJI Interview Insights & Advanced Locks

This article combines a DJI interview experience—including salary and bonus details—with an in‑depth guide to Java concurrency tools, lock implementations, SpringBoot AOP, filters vs. interceptors, Kafka ordering, zero‑copy techniques, and related backend development best practices.

Backend DevelopmentKafkaLocks
0 likes · 26 min read
Master Java Concurrency & SpringBoot: DJI Interview Insights & Advanced Locks
Top Architect
Top Architect
Jan 19, 2025 · Backend Development

Understanding Kafka Consumer: Offset Management, Rebalance, Partition Assignment, and Thread Safety

This article provides a comprehensive technical walkthrough of KafkaConsumer, covering Java configuration code, delivery semantics (at‑most‑once, at‑least‑once, exactly‑once), offset commit strategies, rebalance mechanisms, partition assignment algorithms, thread‑safety concerns, and internal poll implementation, followed by unrelated promotional content.

Backend DevelopmentConsumerKafka
0 likes · 16 min read
Understanding Kafka Consumer: Offset Management, Rebalance, Partition Assignment, and Thread Safety
IT Architects Alliance
IT Architects Alliance
Jan 9, 2025 · Backend Development

Understanding Message Queues: Comparing Kafka, RabbitMQ, and RocketMQ

Message queues act as asynchronous communication bridges in distributed systems, and this article examines three leading solutions—Kafka, RabbitMQ, and RocketMQ—detailing their performance, scalability, reliability, strengths, drawbacks, and suitable use‑cases to guide developers in selecting the right technology.

KafkaMessage QueueRabbitMQ
0 likes · 13 min read
Understanding Message Queues: Comparing Kafka, RabbitMQ, and RocketMQ
dbaplus Community
dbaplus Community
Jan 5, 2025 · Big Data

How DeWu Halved Observability Costs Using AutoMQ and ClickHouse Storage‑Compute Separation

DeWu’s observability platform faced scalability, cost, and operational challenges from petabyte‑scale trace data, prompting a shift to a storage‑compute separated architecture that leverages AutoMQ’s Kafka‑compatible service and ClickHouse Enterprise’s SharedMergeTree engine, ultimately achieving up to 50% cost reduction and five‑fold cold‑read performance gains.

AutoMQClickHouseCost Reduction
0 likes · 20 min read
How DeWu Halved Observability Costs Using AutoMQ and ClickHouse Storage‑Compute Separation
IT Services Circle
IT Services Circle
Jan 3, 2025 · Backend Development

Eight Common Use Cases of Message Queues (MQ) with Code Examples

This article explains eight typical scenarios for using message queues—including asynchronous processing, service decoupling, traffic shaping, delayed tasks, log collection, distributed transactions, remote invocation, and broadcast notifications—providing clear explanations and Java code snippets for each case.

Asynchronous ProcessingKafkaMessage Queue
0 likes · 13 min read
Eight Common Use Cases of Message Queues (MQ) with Code Examples
Ctrip Technology
Ctrip Technology
Jan 3, 2025 · Big Data

Design and Implementation of a Kafka Gatekeeper for FinOps Billing Data Quality Governance

This article describes the challenges of data quality in Ctrip’s hybrid‑cloud FinOps billing system and presents the design, implementation, and high‑availability deployment of a custom Kafka Gatekeeper proxy that performs pre‑validation, configurable rules, self‑service dashboards, and automated alerts to improve coverage, timeliness, and responsibility attribution.

Cloud NativeFinOpsKafka
0 likes · 17 min read
Design and Implementation of a Kafka Gatekeeper for FinOps Billing Data Quality Governance
Programmer XiaoFu
Programmer XiaoFu
Dec 26, 2024 · Backend Development

Avoid Misusing Spring Event: 6 Hard‑Earned Best Practices from a Real‑World Failure

The article recounts a production incident caused by improper use of Spring Event, explains why graceful shutdown, correct startup timing, suitable business scenarios, and reliability measures such as retries and idempotency are essential, and presents six concrete best‑practice recommendations to prevent similar failures.

Backend DevelopmentIdempotencyKafka
0 likes · 11 min read
Avoid Misusing Spring Event: 6 Hard‑Earned Best Practices from a Real‑World Failure
Bin's Tech Cabin
Bin's Tech Cabin
Dec 25, 2024 · Backend Development

Unveiling Netty and Kafka Time Wheels: High‑Performance Scheduling Explained

This article explores the design and implementation of time wheel algorithms in Netty and Kafka, comparing their single‑layer and multi‑layer approaches, analyzing performance trade‑offs, and detailing how these systems achieve O(1) scheduling for massive delayed tasks while avoiding empty ticks.

BackendKafkaNetty
0 likes · 76 min read
Unveiling Netty and Kafka Time Wheels: High‑Performance Scheduling Explained
MaGe Linux Operations
MaGe Linux Operations
Dec 14, 2024 · Big Data

Master Kafka: From Core Concepts to Real-World Deployment

This comprehensive guide explains Kafka’s architecture, core APIs, topics and partitions, deployment steps, multi‑broker clustering, and practical use cases such as messaging, log aggregation, stream processing, and data import/export with Kafka Connect, providing a hands‑on tutorial for developers and engineers.

KafkaTutorialdistributed systems
0 likes · 30 min read
Master Kafka: From Core Concepts to Real-World Deployment
Top Architect
Top Architect
Dec 12, 2024 · Backend Development

Using Spring Transaction Hooks to Send Kafka Messages After Transaction Commit

This article explains how to leverage Spring's TransactionSynchronizationManager to detect active transactions, register synchronization callbacks, and asynchronously send Kafka messages only after a transaction successfully commits, illustrated with a payment‑system example and complete Java code snippets, while also noting thread‑local considerations and promotional offers.

JavaKafkaSpring
0 likes · 12 min read
Using Spring Transaction Hooks to Send Kafka Messages After Transaction Commit
JD Cloud Developers
JD Cloud Developers
Dec 11, 2024 · Operations

How to Prevent Message Queue Disorder in Distributed Systems

This article examines the causes of message queue disorder in distributed systems and presents practical solutions such as ordered messaging, pre‑consumption checks, state machines, and monitoring to improve system stability and data consistency.

KafkaMessage QueueRocketMQ
0 likes · 9 min read
How to Prevent Message Queue Disorder in Distributed Systems
Alibaba Cloud Big Data AI Platform
Alibaba Cloud Big Data AI Platform
Dec 9, 2024 · Big Data

Why Kafka Falls Short for Real‑Time Analytics and How Fluss Changes the Game

Flink Forward Asia 2024 highlighted the limitations of Kafka for real‑time analytics—lack of updates, poor data exploration, costly back‑tracking, and high network overhead—while introducing Fluss, a columnar streaming storage that offers low‑latency reads, CDC, lake‑stream integration, and efficient Delta Join for scalable, fast analytics.

Delta JoinFlinkKafka
0 likes · 15 min read
Why Kafka Falls Short for Real‑Time Analytics and How Fluss Changes the Game
Xiaohongshu Tech REDtech
Xiaohongshu Tech REDtech
Dec 5, 2024 · Big Data

Interview with Jianchen: Journey from Open Source Contributor to Data Engineer at Xiaohongshu

In this interview, Xiaohongshu data engineer Jianchen recounts his evolution from a computer‑science student discovering open‑source through MIT6.824 to contributing to SOFAJRaft and Apache RocketMQ, detailing his OSPP projects, the decision to join Xiaohongshu, and his work on a cloud‑native Kafka engine that cut storage and compute usage by half.

Apache RocketMQCareer DevelopmentCloud Native
0 likes · 11 min read
Interview with Jianchen: Journey from Open Source Contributor to Data Engineer at Xiaohongshu
Java Architect Essentials
Java Architect Essentials
Dec 4, 2024 · Backend Development

Spring Boot Project Initialization, Version Management, and Common Scaffold Code Guide

This article walks through initializing a Spring Boot project, handling version compatibility between Spring Cloud, Spring Boot, and Kafka, demonstrates common scaffolding code such as global exception handling, logging, CORS configuration, and recommends useful tools like Maven, embedded Redis, MyBatis‑Plus, and MapStruct for backend development.

Backend DevelopmentJavaKafka
0 likes · 9 min read
Spring Boot Project Initialization, Version Management, and Common Scaffold Code Guide
Sanyou's Java Diary
Sanyou's Java Diary
Dec 2, 2024 · Big Data

Understanding Kafka: Core Architecture, Storage, and Reliability Explained

This article provides a comprehensive overview of Kafka, covering its overall structure, key components such as brokers, producers, consumers, topics, partitions, replicas, leader‑follower mechanics, logical and physical storage models, producer and consumer workflows, configuration parameters, partition assignment strategies, rebalancing, log retention and compaction, indexing, zero‑copy transmission, and the reliability concepts that ensure data durability.

Data StreamingKafkaMessage Queue
0 likes · 18 min read
Understanding Kafka: Core Architecture, Storage, and Reliability Explained
Top Architect
Top Architect
Nov 25, 2024 · Backend Development

Using Spring Transaction Hooks to Send Kafka Messages After Transaction Commit

This article explains how to leverage Spring's TransactionSynchronizationManager to detect active transactions and register synchronization callbacks so that Kafka messages are sent asynchronously only after a transaction successfully commits, ensuring data consistency without impacting the main business flow.

JavaKafkaSpring
0 likes · 12 min read
Using Spring Transaction Hooks to Send Kafka Messages After Transaction Commit
Top Architect
Top Architect
Nov 20, 2024 · Big Data

Understanding Distributed Systems and Kafka: Architecture, Message Ordering, and Java Consumer Practices

This article explains the fundamentals of distributed systems, introduces Apache Kafka's architecture and components, discusses how Kafka ensures ordered message consumption, and provides Java consumer configuration tips to maintain message order, offering practical guidance for backend developers working with streaming data.

JavaKafkabig data
0 likes · 11 min read
Understanding Distributed Systems and Kafka: Architecture, Message Ordering, and Java Consumer Practices
Programmer1970
Programmer1970
Nov 18, 2024 · Backend Development

How to Resolve Message Backlog and Why Kafka Outperforms RocketMQ

The article outlines practical solutions for RocketMQ message backlog—including scaling consumers, optimizing logic, batch consumption, producer throttling, broker tuning, monitoring, and emergency plans—then explains why Kafka delivers higher performance through sendfile zero‑copy, simple log storage, efficient memory use, and batch sending, and finally compares the two systems on reliability, retry, transactions, filtering, ordering, and suitable use cases.

BacklogKafkaMessage Queue
0 likes · 8 min read
How to Resolve Message Backlog and Why Kafka Outperforms RocketMQ
Huolala Tech
Huolala Tech
Nov 14, 2024 · Operations

How Huolala Scaled Kafka: From Integrated Design to Cloud‑Native Elastic Architecture

This article chronicles the evolution of Huolala’s Kafka infrastructure—from an integrated compute‑storage design to a separated compute‑storage model with multi‑tenant deployment, and finally to a cloud‑native elastic architecture—detailing the challenges of capacity awareness, alarm configuration, and cost‑effective performance optimization.

KafkaOperationscapacity planning
0 likes · 9 min read
How Huolala Scaled Kafka: From Integrated Design to Cloud‑Native Elastic Architecture
Tencent Cloud Developer
Tencent Cloud Developer
Nov 14, 2024 · Cloud Native

ZooKeeper Core Knowledge and Typical Application Scenarios

Although many platforms are dropping ZooKeeper, this guide explains its CP‑oriented architecture, znode structure, watcher mechanism, Zab consensus, leader election, and common patterns such as publish/subscribe, load balancing, naming, master election, distributed locks and queues, giving architects essential fundamentals for coordination services.

Consensus AlgorithmKafkaZAB protocol
0 likes · 25 min read
ZooKeeper Core Knowledge and Typical Application Scenarios
macrozheng
macrozheng
Nov 12, 2024 · Backend Development

Boost Kafka Throughput in Spring Boot: Batch Consumption Guide

This article demonstrates how to integrate Kafka with Spring Boot, add necessary dependencies and configuration, implement both single‑message and batch consumers, and tune batch settings to dramatically improve processing speed for millions of records in a microservice environment.

BackendBatchProcessingJava
0 likes · 12 min read
Boost Kafka Throughput in Spring Boot: Batch Consumption Guide
JD Tech
JD Tech
Nov 7, 2024 · Big Data

JDQ Kafka Bandwidth Control and Rate Limiting Architecture

This article analyzes the limitations of native Apache Kafka rate‑limiting, introduces JDQ's multi‑dimensional, dynamic, priority‑aware bandwidth control framework, and demonstrates its effectiveness through partition‑level, single‑machine, and hierarchical elastic throttling experiments for large‑scale e‑commerce data streams.

Elastic ThrottlingKafkaRate Limiting
0 likes · 18 min read
JDQ Kafka Bandwidth Control and Rate Limiting Architecture
Sohu Tech Products
Sohu Tech Products
Nov 6, 2024 · Operations

Design and Implementation of a Business Operation Log Management System Using Canal and Elasticsearch

The article presents a decoupled business operation log management architecture that uses Alibaba’s Canal to capture MySQL binlog changes, streams them through Kafka, and stores structured before‑and‑after records in Elasticsearch with nested mappings, enabling multi‑table correlation via transaction IDs, visual querying, and reliable rollback without modifying application code.

BackendCanalData Pipeline
0 likes · 12 min read
Design and Implementation of a Business Operation Log Management System Using Canal and Elasticsearch
Mingyi World Elasticsearch
Mingyi World Elasticsearch
Nov 3, 2024 · Big Data

How to Build a Scalable Business Operation Log System with Canal and Elasticsearch

This article walks through the design and implementation of a decoupled, high‑performance business operation log solution that captures MySQL binlog changes via Canal, streams them through Kafka, and stores and queries them in Elasticsearch, addressing challenges such as batch operations, multi‑table transactions, and non‑business data filtering.

BinlogCanalData Pipeline
0 likes · 13 min read
How to Build a Scalable Business Operation Log System with Canal and Elasticsearch
Top Architect
Top Architect
Nov 1, 2024 · Backend Development

Using Spring Transaction Hooks with TransactionSynchronizationManager for Asynchronous Kafka Messaging

The article explains how to leverage Spring's TransactionSynchronizationManager to detect active transactions and register synchronization callbacks that asynchronously send Kafka messages after a transaction commits, ensuring reliable fund‑flow archiving in payment systems while avoiding thread‑switch issues.

JavaKafkaSpring
0 likes · 11 min read
Using Spring Transaction Hooks with TransactionSynchronizationManager for Asynchronous Kafka Messaging
JD Retail Technology
JD Retail Technology
Oct 31, 2024 · Big Data

JDQ Kafka Bandwidth Throttling Architecture and Optimization

This article presents an in‑depth analysis of Kafka's native throttling mechanisms, identifies their limitations in large‑scale e‑commerce scenarios, and introduces JDQ's multi‑dimensional, dynamic throttling architecture that ensures stable throughput and priority‑aware bandwidth management across broker failures and traffic spikes.

JDQKafkabandwidth management
0 likes · 17 min read
JDQ Kafka Bandwidth Throttling Architecture and Optimization
Architect
Architect
Oct 30, 2024 · Backend Development

How to Build Distributed WebSocket Messaging with Spring, Redis, and Kafka

This article explains how to enable cross‑node WebSocket communication in a distributed Spring application by using a message queue (Redis or Kafka) to broadcast messages, tracking user connections with Redis, and providing a complete step‑by‑step implementation with code samples and configuration details.

JavaKafkaRedis
0 likes · 20 min read
How to Build Distributed WebSocket Messaging with Spring, Redis, and Kafka
Tencent Cloud Middleware
Tencent Cloud Middleware
Oct 30, 2024 · Backend Development

How Kafka Guarantees High Reliability and Performance – A Deep Technical Dive

This article thoroughly examines Apache Kafka’s architecture, covering its macro components, ack strategies, replication mechanisms, high‑watermark handling, leader election, and performance optimizations such as batch sending, compression, PageCache, zero‑copy, mmap and sendfile, while also explaining common pitfalls like data loss and log corruption.

KafkaMessage QueuePerformance
0 likes · 31 min read
How Kafka Guarantees High Reliability and Performance – A Deep Technical Dive
Rare Earth Juejin Tech Community
Rare Earth Juejin Tech Community
Oct 21, 2024 · Backend Development

Design and Implementation of a Live Streaming Danmaku System Using Redis, Kafka, and WebSocket

This article explains how to design a live‑streaming danmaku (bullet‑screen) feature by using Redis Zset as the underlying data structure, limiting the list to the latest ten messages, and handling retrieval via polling APIs or WebSocket, with Kafka for message buffering and a distributed lock to avoid ordering issues.

KafkaReal-time MessagingRedis
0 likes · 8 min read
Design and Implementation of a Live Streaming Danmaku System Using Redis, Kafka, and WebSocket
ITPUB
ITPUB
Oct 19, 2024 · Backend Development

Why Some Teams Choose Redis Over Kafka for Message Queuing

The article examines the technical reasons a new tech director prefers using Redis Streams as a lightweight message queue instead of Kafka, comparing their features, operational costs, performance characteristics, and suitability for small‑to‑medium projects.

Backend DevelopmentKafka
0 likes · 7 min read
Why Some Teams Choose Redis Over Kafka for Message Queuing
Top Architect
Top Architect
Oct 18, 2024 · Backend Development

Using Spring Transaction Hooks to Send Kafka Messages After Commit

This article explains how to leverage Spring's TransactionSynchronizationManager to detect active transactions, register synchronization callbacks, and asynchronously send Kafka messages only after a transaction successfully commits, providing a practical example and detailed code snippets for building a reusable Spring Boot starter library.

JavaKafkaSpring
0 likes · 12 min read
Using Spring Transaction Hooks to Send Kafka Messages After Commit
Architect
Architect
Oct 16, 2024 · Backend Development

Mastering Spring Transaction Hooks for Safe Kafka Publishing

This article explains how to use Spring's TransactionSynchronizationManager to detect active transactions, register synchronization callbacks, and asynchronously publish payment‑log messages to Kafka without affecting the main business flow, complete with practical code snippets and pitfalls to avoid.

BackendJavaKafka
0 likes · 11 min read
Mastering Spring Transaction Hooks for Safe Kafka Publishing
Sanyou's Java Diary
Sanyou's Java Diary
Oct 10, 2024 · Big Data

Why Kafka Handles Millions of Messages Per Second: Inside Its High‑Performance Architecture

This article explains how Kafka achieves ultra‑high throughput and low latency despite being disk‑based, covering its Reactor I/O network model, zero‑copy techniques, partitioning strategies, segment logs with sparse indexes, sequential disk writes, page cache usage, compression, batch processing, and lock‑free offset management.

KafkaPerformanceReactor I/O
0 likes · 27 min read
Why Kafka Handles Millions of Messages Per Second: Inside Its High‑Performance Architecture
Sanyou's Java Diary
Sanyou's Java Diary
Sep 19, 2024 · Operations

Message Queue Showdown: Choosing Kafka, Pulsar, RabbitMQ, RocketMQ or NSQ

This comprehensive guide compares five popular message queue systems—Kafka, Pulsar, RabbitMQ, RocketMQ, and NSQ—covering their architectures, core concepts, selection criteria, feature differences such as ordering, latency, durability, scaling, and operational considerations, to help engineers select the most suitable solution for their use cases.

KafkaMessage QueueNSQ
0 likes · 46 min read
Message Queue Showdown: Choosing Kafka, Pulsar, RabbitMQ, RocketMQ or NSQ
Code Ape Tech Column
Code Ape Tech Column
Sep 12, 2024 · Backend Development

Distributed WebSocket Messaging with Redis and Kafka in Spring

This article explains how to enable cross‑server WebSocket communication in a distributed Spring application by using a message queue such as Redis or Kafka, detailing two architectural approaches, concrete code examples, configuration files, and a sample client page.

BackendKafkaMessage Queue
0 likes · 17 min read
Distributed WebSocket Messaging with Redis and Kafka in Spring
MaGe Linux Operations
MaGe Linux Operations
Sep 10, 2024 · Backend Development

How Kafka Elects Leaders and Distributes Partitions: A Deep Dive

This article explains Kafka's leader election process, partition assignment strategy, distribution policies, file layout, and the evolution of consumer offset storage, providing a comprehensive overview of how Kafka ensures reliable and efficient message handling in a distributed environment.

KafkaPartition Assignmentdistributed systems
0 likes · 5 min read
How Kafka Elects Leaders and Distributes Partitions: A Deep Dive
Tencent Cloud Developer
Tencent Cloud Developer
Sep 6, 2024 · Backend Development

Message Queue (MQ) Overview, Selection Criteria, and Comparison of Kafka, Pulsar, RocketMQ, RabbitMQ, and NSQ

The article surveys major message‑queue systems—Kafka, Pulsar, RocketMQ, RabbitMQ, and NSQ—detailing their architectures, key features, selection criteria such as ordering, scalability and latency, performance benchmarks, operational considerations, and typical use cases, and advises which platform best fits high‑throughput, reliability, low‑latency, or lightweight Go‑centric scenarios.

Backend ArchitectureKafkaNSQ
0 likes · 47 min read
Message Queue (MQ) Overview, Selection Criteria, and Comparison of Kafka, Pulsar, RocketMQ, RabbitMQ, and NSQ
Top Architect
Top Architect
Sep 3, 2024 · Backend Development

Design and Implementation of a General Asynchronous Processing SDK for Java Backend

This article introduces a reusable asynchronous processing SDK for Java backend systems, explaining its purpose, advantages, underlying principles, component choices, design patterns, database schema, configuration options, usage instructions, and best‑practice notes, while providing complete code snippets and deployment details.

BackendDesign PatternKafka
0 likes · 13 min read
Design and Implementation of a General Asynchronous Processing SDK for Java Backend
JavaEdge
JavaEdge
Sep 1, 2024 · Backend Development

How Kafka Implements Transactions and Achieves Exactly‑Once Guarantees

This article explains how Kafka implements transactions using a two‑phase commit protocol, compares its approach with RocketMQ, details the exactly‑once semantics for streaming workloads, and walks through the coordinator roles, message flow, and practical use‑case scenarios.

KafkaTransactiontwo-phase commit
0 likes · 10 min read
How Kafka Implements Transactions and Achieves Exactly‑Once Guarantees
21CTO
21CTO
Aug 28, 2024 · Backend Development

Why Message Queues Are Essential for Scalable Backend Systems

This article explains how message queues decouple and buffer tasks such as payment processing, inventory updates, and notifications in high‑traffic online stores, improving reliability, scalability, and fault tolerance through asynchronous communication.

Asynchronous ProcessingBackend ArchitectureKafka
0 likes · 6 min read
Why Message Queues Are Essential for Scalable Backend Systems
Su San Talks Tech
Su San Talks Tech
Aug 26, 2024 · Backend Development

7 Real-World Message Queue Patterns Every Backend Engineer Should Know

This article shares seven classic message‑queue use cases—from asynchronous decoupling and peak‑smoothing to delayed tasks, broadcast consumption, distributed transactions, and data hub integration—illustrated with real‑world examples, diagrams, and RocketMQ code snippets.

KafkaRocketMQ
0 likes · 12 min read
7 Real-World Message Queue Patterns Every Backend Engineer Should Know
Architect's Journey
Architect's Journey
Aug 19, 2024 · Backend Development

Flexible Multi-MQ Component Update: Switch Between RocketMQ, Redis, Kafka, and RabbitMQ

The article explains why adopting a message‑queue improves microservice decoupling and responsibility boundaries, then details the design of a pluggable Base‑MQ component—including @MQEventListener, MQEvent base class, configuration, and MQClient interface—followed by concrete implementations for RocketMQ, Kafka, Redis and RabbitMQ, an example event flow, and guidance for extending the framework.

JavaKafkaMessage Queue
0 likes · 27 min read
Flexible Multi-MQ Component Update: Switch Between RocketMQ, Redis, Kafka, and RabbitMQ
Hacker Afternoon Tea
Hacker Afternoon Tea
Aug 16, 2024 · Backend Development

Building a Fast Real‑Time Streaming Platform with Kafka and ClickHouse

This guide walks through the challenges of ingesting high‑volume sales‑point data into ClickHouse, explains why a buffering layer like Kafka is essential, compares three integration approaches—ClickHouse Kafka engine, Kafka Connect, and an external writer—and provides concrete SQL and Terraform examples to help you choose the right solution.

ClickHouseData PipelineExternal writer
0 likes · 13 min read
Building a Fast Real‑Time Streaming Platform with Kafka and ClickHouse