BestHub
Sign in
Tagged articles
1273 articles
Page 3 of 13
Top Architect
Top Architect
Nov 20, 2024 · Big Data

Understanding Distributed Systems and Kafka: Architecture, Message Ordering, and Java Consumer Practices

This article explains the fundamentals of distributed systems, introduces Apache Kafka's architecture and components, discusses how Kafka ensures ordered message consumption, and provides Java consumer configuration tips to maintain message order, offering practical guidance for backend developers working with streaming data.

Big DataDistributed SystemsKafka
0 likes · 11 min read
Understanding Distributed Systems and Kafka: Architecture, Message Ordering, and Java Consumer Practices
Huolala Tech
Huolala Tech
Nov 14, 2024 · Operations

How Huolala Scaled Kafka: From Integrated Design to Cloud‑Native Elastic Architecture

This article chronicles the evolution of Huolala’s Kafka infrastructure—from an integrated compute‑storage design to a separated compute‑storage model with multi‑tenant deployment, and finally to a cloud‑native elastic architecture—detailing the challenges of capacity awareness, alarm configuration, and cost‑effective performance optimization.

KafkaOperationscapacity planning
0 likes · 9 min read
How Huolala Scaled Kafka: From Integrated Design to Cloud‑Native Elastic Architecture
Tencent Cloud Developer
Tencent Cloud Developer
Nov 14, 2024 · Cloud Native

ZooKeeper Core Knowledge and Typical Application Scenarios

Although many platforms are dropping ZooKeeper, this guide explains its CP‑oriented architecture, znode structure, watcher mechanism, Zab consensus, leader election, and common patterns such as publish/subscribe, load balancing, naming, master election, distributed locks and queues, giving architects essential fundamentals for coordination services.

Consensus AlgorithmDistributed CoordinationKafka
0 likes · 25 min read
ZooKeeper Core Knowledge and Typical Application Scenarios
macrozheng
macrozheng
Nov 12, 2024 · Backend Development

Boost Kafka Throughput in Spring Boot: Batch Consumption Guide

This article demonstrates how to integrate Kafka with Spring Boot, add necessary dependencies and configuration, implement both single‑message and batch consumers, and tune batch settings to dramatically improve processing speed for millions of records in a microservice environment.

BackendBatchProcessingKafka
0 likes · 12 min read
Boost Kafka Throughput in Spring Boot: Batch Consumption Guide
JD Tech
JD Tech
Nov 7, 2024 · Big Data

JDQ Kafka Bandwidth Control and Rate Limiting Architecture

This article analyzes the limitations of native Apache Kafka rate‑limiting, introduces JDQ's multi‑dimensional, dynamic, priority‑aware bandwidth control framework, and demonstrates its effectiveness through partition‑level, single‑machine, and hierarchical elastic throttling experiments for large‑scale e‑commerce data streams.

Distributed StreamingElastic ThrottlingKafka
0 likes · 18 min read
JDQ Kafka Bandwidth Control and Rate Limiting Architecture
Sohu Tech Products
Sohu Tech Products
Nov 6, 2024 · Operations

Design and Implementation of a Business Operation Log Management System Using Canal and Elasticsearch

The article presents a decoupled business operation log management architecture that uses Alibaba’s Canal to capture MySQL binlog changes, streams them through Kafka, and stores structured before‑and‑after records in Elasticsearch with nested mappings, enabling multi‑table correlation via transaction IDs, visual querying, and reliable rollback without modifying application code.

BackendCanalElasticsearch
0 likes · 12 min read
Design and Implementation of a Business Operation Log Management System Using Canal and Elasticsearch
Mingyi World Elasticsearch
Mingyi World Elasticsearch
Nov 3, 2024 · Big Data

How to Build a Scalable Business Operation Log System with Canal and Elasticsearch

This article walks through the design and implementation of a decoupled, high‑performance business operation log solution that captures MySQL binlog changes via Canal, streams them through Kafka, and stores and queries them in Elasticsearch, addressing challenges such as batch operations, multi‑table transactions, and non‑business data filtering.

BinlogCanalElasticsearch
0 likes · 13 min read
How to Build a Scalable Business Operation Log System with Canal and Elasticsearch
Top Architect
Top Architect
Nov 1, 2024 · Backend Development

Using Spring Transaction Hooks with TransactionSynchronizationManager for Asynchronous Kafka Messaging

The article explains how to leverage Spring's TransactionSynchronizationManager to detect active transactions and register synchronization callbacks that asynchronously send Kafka messages after a transaction commits, ensuring reliable fund‑flow archiving in payment systems while avoiding thread‑switch issues.

Kafkajavaspring
0 likes · 11 min read
Using Spring Transaction Hooks with TransactionSynchronizationManager for Asynchronous Kafka Messaging
JD Retail Technology
JD Retail Technology
Oct 31, 2024 · Big Data

JDQ Kafka Bandwidth Throttling Architecture and Optimization

This article presents an in‑depth analysis of Kafka's native throttling mechanisms, identifies their limitations in large‑scale e‑commerce scenarios, and introduces JDQ's multi‑dimensional, dynamic throttling architecture that ensures stable throughput and priority‑aware bandwidth management across broker failures and traffic spikes.

Distributed SystemsJDQKafka
0 likes · 17 min read
JDQ Kafka Bandwidth Throttling Architecture and Optimization
Architect
Architect
Oct 30, 2024 · Backend Development

How to Build Distributed WebSocket Messaging with Spring, Redis, and Kafka

This article explains how to enable cross‑node WebSocket communication in a distributed Spring application by using a message queue (Redis or Kafka) to broadcast messages, tracking user connections with Redis, and providing a complete step‑by‑step implementation with code samples and configuration details.

Distributed SystemsKafkaWebSocket
0 likes · 20 min read
How to Build Distributed WebSocket Messaging with Spring, Redis, and Kafka
Tencent Cloud Middleware
Tencent Cloud Middleware
Oct 30, 2024 · Backend Development

How Kafka Guarantees High Reliability and Performance – A Deep Technical Dive

This article thoroughly examines Apache Kafka’s architecture, covering its macro components, ack strategies, replication mechanisms, high‑watermark handling, leader election, and performance optimizations such as batch sending, compression, PageCache, zero‑copy, mmap and sendfile, while also explaining common pitfalls like data loss and log corruption.

Distributed SystemsKafkaMessage Queue
0 likes · 31 min read
How Kafka Guarantees High Reliability and Performance – A Deep Technical Dive
Rare Earth Juejin Tech Community
Rare Earth Juejin Tech Community
Oct 21, 2024 · Backend Development

Design and Implementation of a Live Streaming Danmaku System Using Redis, Kafka, and WebSocket

This article explains how to design a live‑streaming danmaku (bullet‑screen) feature by using Redis Zset as the underlying data structure, limiting the list to the latest ten messages, and handling retrieval via polling APIs or WebSocket, with Kafka for message buffering and a distributed lock to avoid ordering issues.

KafkaWebSocketdanmaku
0 likes · 8 min read
Design and Implementation of a Live Streaming Danmaku System Using Redis, Kafka, and WebSocket
ITPUB
ITPUB
Oct 19, 2024 · Backend Development

Why Some Teams Choose Redis Over Kafka for Message Queuing

The article examines the technical reasons a new tech director prefers using Redis Streams as a lightweight message queue instead of Kafka, comparing their features, operational costs, performance characteristics, and suitability for small‑to‑medium projects.

Kafkabackend-development
0 likes · 7 min read
Why Some Teams Choose Redis Over Kafka for Message Queuing
Top Architect
Top Architect
Oct 18, 2024 · Backend Development

Using Spring Transaction Hooks to Send Kafka Messages After Commit

This article explains how to leverage Spring's TransactionSynchronizationManager to detect active transactions, register synchronization callbacks, and asynchronously send Kafka messages only after a transaction successfully commits, providing a practical example and detailed code snippets for building a reusable Spring Boot starter library.

KafkaSpringBootTransactionSynchronizationManager
0 likes · 12 min read
Using Spring Transaction Hooks to Send Kafka Messages After Commit
Architect
Architect
Oct 16, 2024 · Backend Development

Mastering Spring Transaction Hooks for Safe Kafka Publishing

This article explains how to use Spring's TransactionSynchronizationManager to detect active transactions, register synchronization callbacks, and asynchronously publish payment‑log messages to Kafka without affecting the main business flow, complete with practical code snippets and pitfalls to avoid.

BackendKafkaTransactionSynchronizationManager
0 likes · 11 min read
Mastering Spring Transaction Hooks for Safe Kafka Publishing
Sanyou's Java Diary
Sanyou's Java Diary
Oct 10, 2024 · Big Data

Why Kafka Handles Millions of Messages Per Second: Inside Its High‑Performance Architecture

This article explains how Kafka achieves ultra‑high throughput and low latency despite being disk‑based, covering its Reactor I/O network model, zero‑copy techniques, partitioning strategies, segment logs with sparse indexes, sequential disk writes, page cache usage, compression, batch processing, and lock‑free offset management.

KafkaOffset ManagementReactor I/O
0 likes · 27 min read
Why Kafka Handles Millions of Messages Per Second: Inside Its High‑Performance Architecture
Sanyou's Java Diary
Sanyou's Java Diary
Sep 19, 2024 · Operations

Message Queue Showdown: Choosing Kafka, Pulsar, RabbitMQ, RocketMQ or NSQ

This comprehensive guide compares five popular message queue systems—Kafka, Pulsar, RabbitMQ, RocketMQ, and NSQ—covering their architectures, core concepts, selection criteria, feature differences such as ordering, latency, durability, scaling, and operational considerations, to help engineers select the most suitable solution for their use cases.

ComparisonKafkaMessage Queue
0 likes · 46 min read
Message Queue Showdown: Choosing Kafka, Pulsar, RabbitMQ, RocketMQ or NSQ
Code Ape Tech Column
Code Ape Tech Column
Sep 14, 2024 · Backend Development

Using Spring TransactionSynchronizationManager for Transaction Hooks to Send Kafka Messages

This article demonstrates how to leverage Spring's TransactionSynchronizationManager to detect active transactions and register synchronization callbacks that asynchronously publish payment ledger messages to Kafka after transaction commit, while providing a starter-style library for easy integration.

BackendKafkaTransactionSynchronizationManager
0 likes · 9 min read
Using Spring TransactionSynchronizationManager for Transaction Hooks to Send Kafka Messages
Code Ape Tech Column
Code Ape Tech Column
Sep 12, 2024 · Backend Development

Distributed WebSocket Messaging with Redis and Kafka in Spring

This article explains how to enable cross‑server WebSocket communication in a distributed Spring application by using a message queue such as Redis or Kafka, detailing two architectural approaches, concrete code examples, configuration files, and a sample client page.

BackendDistributedKafka
0 likes · 17 min read
Distributed WebSocket Messaging with Redis and Kafka in Spring
MaGe Linux Operations
MaGe Linux Operations
Sep 10, 2024 · Backend Development

How Kafka Elects Leaders and Distributes Partitions: A Deep Dive

This article explains Kafka's leader election process, partition assignment strategy, distribution policies, file layout, and the evolution of consumer offset storage, providing a comprehensive overview of how Kafka ensures reliable and efficient message handling in a distributed environment.

Distributed SystemsKafkaPartition Assignment
0 likes · 5 min read
How Kafka Elects Leaders and Distributes Partitions: A Deep Dive
Tencent Cloud Developer
Tencent Cloud Developer
Sep 6, 2024 · Backend Development

Message Queue (MQ) Overview, Selection Criteria, and Comparison of Kafka, Pulsar, RocketMQ, RabbitMQ, and NSQ

The article surveys major message‑queue systems—Kafka, Pulsar, RocketMQ, RabbitMQ, and NSQ—detailing their architectures, key features, selection criteria such as ordering, scalability and latency, performance benchmarks, operational considerations, and typical use cases, and advises which platform best fits high‑throughput, reliability, low‑latency, or lightweight Go‑centric scenarios.

Backend ArchitectureKafkaNSQ
0 likes · 47 min read
Message Queue (MQ) Overview, Selection Criteria, and Comparison of Kafka, Pulsar, RocketMQ, RabbitMQ, and NSQ
Top Architect
Top Architect
Sep 3, 2024 · Backend Development

Design and Implementation of a General Asynchronous Processing SDK for Java Backend

This article introduces a reusable asynchronous processing SDK for Java backend systems, explaining its purpose, advantages, underlying principles, component choices, design patterns, database schema, configuration options, usage instructions, and best‑practice notes, while providing complete code snippets and deployment details.

AsyncBackendKafka
0 likes · 13 min read
Design and Implementation of a General Asynchronous Processing SDK for Java Backend
JavaEdge
JavaEdge
Sep 1, 2024 · Backend Development

How Kafka Implements Transactions and Achieves Exactly‑Once Guarantees

This article explains how Kafka implements transactions using a two‑phase commit protocol, compares its approach with RocketMQ, details the exactly‑once semantics for streaming workloads, and walks through the coordinator roles, message flow, and practical use‑case scenarios.

Kafkatransactiontwo-phase commit
0 likes · 10 min read
How Kafka Implements Transactions and Achieves Exactly‑Once Guarantees
21CTO
21CTO
Aug 28, 2024 · Backend Development

Why Message Queues Are Essential for Scalable Backend Systems

This article explains how message queues decouple and buffer tasks such as payment processing, inventory updates, and notifications in high‑traffic online stores, improving reliability, scalability, and fault tolerance through asynchronous communication.

Backend ArchitectureKafkaMessage Queue
0 likes · 6 min read
Why Message Queues Are Essential for Scalable Backend Systems
Su San Talks Tech
Su San Talks Tech
Aug 26, 2024 · Backend Development

7 Real-World Message Queue Patterns Every Backend Engineer Should Know

This article shares seven classic message‑queue use cases—from asynchronous decoupling and peak‑smoothing to delayed tasks, broadcast consumption, distributed transactions, and data hub integration—illustrated with real‑world examples, diagrams, and RocketMQ code snippets.

KafkaRocketMQ
0 likes · 12 min read
7 Real-World Message Queue Patterns Every Backend Engineer Should Know
Architect's Journey
Architect's Journey
Aug 19, 2024 · Backend Development

Flexible Multi-MQ Component Update: Switch Between RocketMQ, Redis, Kafka, and RabbitMQ

The article explains why adopting a message‑queue improves microservice decoupling and responsibility boundaries, then details the design of a pluggable Base‑MQ component—including @MQEventListener, MQEvent base class, configuration, and MQClient interface—followed by concrete implementations for RocketMQ, Kafka, Redis and RabbitMQ, an example event flow, and guidance for extending the framework.

KafkaMessage QueueMicroservices
0 likes · 27 min read
Flexible Multi-MQ Component Update: Switch Between RocketMQ, Redis, Kafka, and RabbitMQ
Efficient Ops
Efficient Ops
Aug 14, 2024 · Operations

Building a Real-Time Log Monitoring System with ELK, Kafka, and Python

This article details how to construct a log‑monitoring platform using the ELK stack, Kafka buffering, and a Python scheduler to collect, process, and alert on error logs, offering practical configuration tips and performance optimizations for production environments.

ELKElasticsearchKafka
0 likes · 10 min read
Building a Real-Time Log Monitoring System with ELK, Kafka, and Python
StarRocks
StarRocks
Aug 14, 2024 · Big Data

Mastering StarRocks & Apache Paimon: A Fast‑Track Lakehouse Guide

This guide provides a comprehensive overview of Apache Paimon’s architecture, key features, and advantages, explains how to integrate it with StarRocks for real‑time lakehouse analytics, and walks through a complete quick‑start setup including component installation, Flink and Kafka deployment, data ingestion, table creation, and query execution with time‑travel support.

Apache PaimonFlinkKafka
0 likes · 18 min read
Mastering StarRocks & Apache Paimon: A Fast‑Track Lakehouse Guide
Java Interview Crash Guide
Java Interview Crash Guide
Aug 13, 2024 · Backend Development

How to Build a Robust Asynchronous Processing SDK with Spring and Kafka

This article explains the design, advantages, and implementation details of a generic asynchronous processing SDK that leverages Spring transactional events, Kafka messaging, XXL‑Job scheduling, and a non‑intrusive architecture to ensure data consistency, fault tolerance, and easy integration across services.

Design PatternsKafkaasynchronous processing
0 likes · 11 min read
How to Build a Robust Asynchronous Processing SDK with Spring and Kafka
Top Architect
Top Architect
Aug 7, 2024 · Backend Development

Kafka Consumer Deep Dive: Offset Management, Rebalance Strategies, and Thread Safety

This article explains Kafka consumer semantics such as at‑most‑once, at‑least‑once and exactly‑once delivery, shows how to configure and commit offsets, describes the rebalance process and partition‑assignment strategies, and discusses thread‑safety and task scheduling with illustrative Java code examples.

ConsumerKafkaOFFSET
0 likes · 17 min read
Kafka Consumer Deep Dive: Offset Management, Rebalance Strategies, and Thread Safety
Java Backend Technology
Java Backend Technology
Aug 6, 2024 · Backend Development

Master Spring Boot Project Setup: From IDE to Version Management & Common Tools

This guide walks through initializing a Spring Boot project, handling IDE licensing hurdles, ensuring compatible Spring Cloud, Spring Boot, and Kafka versions, leveraging Maven for dependency management, and integrating essential utilities like global exception handling, logging, CORS, Swagger, and helpful development tools.

BackendDevOpsKafka
0 likes · 11 min read
Master Spring Boot Project Setup: From IDE to Version Management & Common Tools
Code Ape Tech Column
Code Ape Tech Column
Aug 5, 2024 · Backend Development

Design and Implementation of a Generic Asynchronous Processing SDK for Java Backend

This article describes a Java‑based asynchronous processing SDK that leverages Spring AOP, transactional event listeners, Kafka, XXL‑Job, and MySQL to provide non‑intrusive, reliable, and eventually consistent background execution of business methods while preserving transaction integrity and offering configurable retry and compensation strategies.

AsyncKafkaaop
0 likes · 8 min read
Design and Implementation of a Generic Asynchronous Processing SDK for Java Backend
Architect
Architect
Aug 3, 2024 · Big Data

Kafka Storage Architecture Design: Deep Analysis and Implementation

This article thoroughly examines Kafka's storage architecture, analyzing its design motivations, storage mechanisms, log formats, partitioning, indexing, cleanup strategies, and performance optimizations such as sequential log writes, sparse indexing, page cache, and zero‑copy, providing insights applicable to other storage systems.

KafkaLog SegmentsSparse Index
0 likes · 22 min read
Kafka Storage Architecture Design: Deep Analysis and Implementation
macrozheng
macrozheng
Aug 2, 2024 · Backend Development

How to Quickly Resolve Massive Kafka Message Backlog in Production

This guide explains why Kafka message backlogs occur, how to diagnose bugs, optimize consumer logic, and use temporary topics for emergency scaling, while emphasizing monitoring, alerts, and proper offset handling to keep your streaming system healthy.

BacklogConsumerKafka
0 likes · 5 min read
How to Quickly Resolve Massive Kafka Message Backlog in Production
Top Architect
Top Architect
Jul 31, 2024 · Backend Development

Design and Implementation of a Generic Asynchronous Processing SDK for Spring Applications

This article introduces a Spring‑based asynchronous processing SDK that leverages annotations, transaction event listeners, Kafka, XXL‑Job, and a dedicated database to achieve non‑blocking execution, fault tolerance, and eventual consistency while preserving transaction integrity and providing a clear configuration and usage guide.

BackendKafkadatabase
0 likes · 12 min read
Design and Implementation of a Generic Asynchronous Processing SDK for Spring Applications
Soul Technical Team
Soul Technical Team
Jul 23, 2024 · Big Data

Kafka Stability Challenges and Governance Framework at Soul

This article analyzes the role, application scenarios, stability challenges, and comprehensive governance framework of Apache Kafka at Soul, covering deployment, configuration, monitoring, standard controls, common misuse, and future directions toward cloud‑native solutions.

KafkaOperationsStreaming
0 likes · 30 min read
Kafka Stability Challenges and Governance Framework at Soul
Rare Earth Juejin Tech Community
Rare Earth Juejin Tech Community
Jul 22, 2024 · Big Data

Comprehensive Guide to Kafka: Architecture, Core Concepts, and Configuration

This article provides an in‑depth overview of Apache Kafka, covering its use cases, comparison with other message queues, versioning, performance mechanisms, core concepts such as topics, partitions, offsets, consumer groups, rebalancing, replication, leader election, idempotence, transactions, compression, interceptors, request handling, and practical configuration tips for reliable streaming applications.

Big DataConsumerKafka
0 likes · 25 min read
Comprehensive Guide to Kafka: Architecture, Core Concepts, and Configuration
Architect
Architect
Jul 12, 2024 · Backend Development

How to Prevent Message Loss and Duplication in Kafka and RocketMQ

This article analyzes the root causes of message loss and duplication in modern message queues, explains how synchronous and asynchronous sending affect reliability, compares storage strategies like sync‑flush and broker clustering, and offers concrete idempotent handling techniques using database constraints and Redis.

BackendKafkaMessage Queue
0 likes · 8 min read
How to Prevent Message Loss and Duplication in Kafka and RocketMQ
Architect's Guide
Architect's Guide
Jul 11, 2024 · Backend Development

Using Spring Transaction Hooks to Send Kafka Messages After Commit

This article demonstrates how to leverage Spring's TransactionSynchronizationManager to detect active transactions and register synchronization callbacks that asynchronously push payment‑ledger messages to Kafka only after the transaction successfully commits, ensuring data consistency and minimal impact on the main business flow.

BackendKafkaTransactionSynchronizationManager
0 likes · 9 min read
Using Spring Transaction Hooks to Send Kafka Messages After Commit
Mike Chen's Internet Architecture
Mike Chen's Internet Architecture
Jul 3, 2024 · Backend Development

Preventing Duplicate Message Consumption in Kafka

This article explains why duplicate message consumption occurs in Kafka, outlines the underlying confirmation and consumer failure issues, and presents idempotent design strategies such as deduplication tables and code examples to ensure reliable processing.

Duplicate ConsumptionKafkaMessage Queue
0 likes · 5 min read
Preventing Duplicate Message Consumption in Kafka
360 Smart Cloud
360 Smart Cloud
Jul 3, 2024 · Operations

Practical Practices for Enhancing Kafka Cluster Stability at 360

This article details 360's comprehensive approach to improving Apache Kafka cluster stability through proactive operations, capacity assessment, parameter tuning, monitoring, version upgrades, and traffic control, offering concrete guidelines and best‑practice recommendations for large‑scale message‑queue deployments.

ClusterKafkaTuning
0 likes · 33 min read
Practical Practices for Enhancing Kafka Cluster Stability at 360
Java Architect Essentials
Java Architect Essentials
Jun 26, 2024 · Databases

Why Organizations Should Consider Using Apache Kafka Instead of Relational Databases

This article explains why organizations may replace traditional relational databases with Apache Kafka as a system of record, highlighting Kafka's economic, scalable, immutable log capabilities, event replay, flexibility for diverse use cases, and its suitability for highly regulated, data‑intensive environments.

Data StreamingEvent-Driven ArchitectureImmutable Log
0 likes · 10 min read
Why Organizations Should Consider Using Apache Kafka Instead of Relational Databases
Architect
Architect
Jun 25, 2024 · Backend Development

Mastering Spring Transaction Hooks: Send Kafka Messages Only After Commit

This article explains how to use Spring's TransactionSynchronizationManager to detect active transactions and register callbacks that asynchronously send Kafka messages after a transaction successfully commits, ensuring data consistency in payment‑system archiving scenarios.

BackendKafkaSpringBoot
0 likes · 10 min read
Mastering Spring Transaction Hooks: Send Kafka Messages Only After Commit
macrozheng
macrozheng
Jun 25, 2024 · Backend Development

7 Real-World Message Queue Patterns Every Backend Engineer Should Master

This article explores seven classic message‑queue use cases—from asynchronous decoupling and traffic‑shaping to delayed tasks, broadcast consumption, distributed transactions, and using Kafka as a data hub—illustrated with real‑world examples and code snippets.

Broadcast ConsumptionDelayed MessageKafka
0 likes · 12 min read
7 Real-World Message Queue Patterns Every Backend Engineer Should Master
Wukong Talks Architecture
Wukong Talks Architecture
Jun 20, 2024 · Backend Development

Seven Classic Use Cases of Message Queues

This article shares seven practical scenarios—such as asynchronous processing, traffic smoothing, message bus, delayed tasks, broadcast consumption, distributed transactions, and data hub integration—where message queues like RocketMQ, Kafka, ActiveMQ, and RabbitMQ help solve high‑concurrency challenges in modern backend systems.

Broadcast ConsumptionDelay MessagesDistributed Systems
0 likes · 11 min read
Seven Classic Use Cases of Message Queues
Sohu Tech Products
Sohu Tech Products
Jun 5, 2024 · Big Data

Why Kafka Is the Backbone of Modern Data Pipelines: Core Architecture and Use Cases

This article explains Kafka's role as a high‑throughput distributed message queue, detailing its core components, topic‑partition model, consumer groups, storage mechanisms, fault‑tolerance features, delivery guarantees, ZooKeeper coordination, and scalability strategies for building reliable real‑time data pipelines.

Big DataDistributed SystemsKafka
0 likes · 14 min read
Why Kafka Is the Backbone of Modern Data Pipelines: Core Architecture and Use Cases
Su San Talks Tech
Su San Talks Tech
Jun 2, 2024 · Big Data

Mastering Kafka: Core Architecture, Use Cases, and Design Principles

This article provides a comprehensive overview of Apache Kafka, covering its role as a message queue, core components, topic and partition design, consumer groups, storage mechanisms, high‑availability features, delivery guarantees, ZooKeeper coordination, and scalability strategies for building robust real‑time data pipelines.

Big DataKafkaStreaming
0 likes · 15 min read
Mastering Kafka: Core Architecture, Use Cases, and Design Principles
Xiaohongshu Tech REDtech
Xiaohongshu Tech REDtech
May 23, 2024 · Cloud Native

Cloud-Native Architecture and Tiered Storage for Xiaohongshu Kafka: Cost Reduction, Elastic Migration, and Performance Optimization

Xiaohongshu's big-data storage team built cloud-native architecture with tiered storage, containerized Kafka, and custom load balancer, cutting storage costs up to 60%, enabling minute‑level elastic migration, improving scaling efficiency tenfold, and boosting performance via caching and batch reads.

Cost OptimizationKafkaelastic scaling
0 likes · 20 min read
Cloud-Native Architecture and Tiered Storage for Xiaohongshu Kafka: Cost Reduction, Elastic Migration, and Performance Optimization
Qunar Tech Salon
Qunar Tech Salon
May 20, 2024 · Big Data

Optimizing Kafka Production at Qunar Travel: Reducing CPU Usage by 2000 Cores

This article presents a comprehensive case study of how Qunar Travel identified and resolved Kafka production bottlenecks—through metric monitoring, thread and flush parameter tuning, and Filebeat batch adjustments—resulting in a 2000‑core CPU reduction, higher network idle rates, and lower resource consumption across three clusters.

Kafkamonitoring
0 likes · 12 min read
Optimizing Kafka Production at Qunar Travel: Reducing CPU Usage by 2000 Cores
Su San Talks Tech
Su San Talks Tech
May 14, 2024 · Backend Development

How We Fixed Repeated Kafka Message Backlog in a High‑Traffic Restaurant System

This article details a series of Kafka message backlog incidents in a busy restaurant ordering system, analyzes root causes such as consumer speed, database indexing, and sudden traffic spikes, and presents step‑by‑step optimizations—including batch queries, index tuning, data archiving, and thread‑pool adjustments—to restore real‑time performance.

Kafkamysqlperformance optimization
0 likes · 11 min read
How We Fixed Repeated Kafka Message Backlog in a High‑Traffic Restaurant System
Mike Chen's Internet Architecture
Mike Chen's Internet Architecture
May 11, 2024 · Big Data

Comprehensive Introduction to Apache Kafka: Architecture, Features, and Use Cases

This article provides a detailed overview of Apache Kafka, covering its core characteristics, distributed architecture, key components such as topics, partitions, brokers, producers, consumers, ZooKeeper, and common application scenarios like log collection, event‑driven architecture, real‑time analytics, and monitoring.

Big DataDistributed SystemsKafka
0 likes · 7 min read
Comprehensive Introduction to Apache Kafka: Architecture, Features, and Use Cases
Lobster Programming
Lobster Programming
May 9, 2024 · Backend Development

Detect Hot Items in High‑Concurrency Sales Using OpenResty, Kafka & Elasticsearch

During flash‑sale events like JD or Taobao, a tiny fraction of products generate the majority of traffic, causing server overload; this guide explains how to identify and collect hotspot product data in real time using an OpenResty‑Lua front‑end, Kafka for messaging, and Elasticsearch for storage and analysis.

ElasticsearchHot ItemsKafka
0 likes · 5 min read
Detect Hot Items in High‑Concurrency Sales Using OpenResty, Kafka & Elasticsearch
Java Tech Enthusiast
Java Tech Enthusiast
Apr 30, 2024 · Backend Development

How to Prevent Message Loss and Duplication in Message Queues

This article examines why messages can be lost or duplicated in typical queue systems, explains the failure points from producer to broker to consumer, and provides practical techniques such as synchronous flushing, broker clustering, database unique constraints, and Redis deduplication to achieve reliable, idempotent processing.

BackendIdempotencyKafka
0 likes · 7 min read
How to Prevent Message Loss and Duplication in Message Queues
Lobster Programming
Lobster Programming
Apr 29, 2024 · Backend Development

Step-by-Step Guide to Installing and Running Kafka on Linux

This article explains what Kafka is, outlines the prerequisites, provides detailed Linux installation steps, configuration tweaks, startup commands, testing procedures, and troubleshooting tips, enabling readers to set up a functional Kafka cluster and produce‑consume messages.

InstallationKafkaLinux
0 likes · 5 min read
Step-by-Step Guide to Installing and Running Kafka on Linux
Architect's Guide
Architect's Guide
Apr 28, 2024 · Backend Development

Kafka Consumer Usage Example and Deep Dive into Offset Management, Rebalance, and Thread Safety

This article presents a Java Kafka consumer example, explains offset semantics (at‑most‑once, at‑least‑once, exactly‑once), details consumer rebalance mechanisms, partition assignment strategies, thread‑safety considerations, and showcases core poll, heartbeat, and auto‑commit implementations with accompanying code snippets.

ConsumerKafkaOffset Management
0 likes · 14 min read
Kafka Consumer Usage Example and Deep Dive into Offset Management, Rebalance, and Thread Safety
Su San Talks Tech
Su San Talks Tech
Apr 25, 2024 · Backend Development

Choosing the Right Message Queue: Kafka, RabbitMQ, RocketMQ, and ActiveMQ Compared

This article provides a comprehensive overview of four major message queue systems—Kafka, RabbitMQ, RocketMQ, and ActiveMQ—covering their fundamentals, architectures, key concepts, advantages, disadvantages, and guidance for selecting the most suitable solution based on throughput, reliability, scalability, and use‑case requirements.

BackendKafkaMessage Queue
0 likes · 18 min read
Choosing the Right Message Queue: Kafka, RabbitMQ, RocketMQ, and ActiveMQ Compared
21CTO
21CTO
Apr 22, 2024 · Big Data

Inside Uber’s Real‑Time Data Infrastructure: How They Scale Streaming at Massive Scale

This article explores Uber’s sophisticated real‑time data infrastructure, detailing how the company leverages open‑source technologies such as Apache Kafka, Flink, Pinot, and Presto, and describing the architectural components, scaling challenges, multi‑region resilience, data back‑filling, and operational practices that enable low‑latency analytics for millions of daily rides and deliveries.

Big DataFlinkKafka
0 likes · 25 min read
Inside Uber’s Real‑Time Data Infrastructure: How They Scale Streaming at Massive Scale
Code Ape Tech Column
Code Ape Tech Column
Apr 18, 2024 · Backend Development

Applying Spring Transaction Hooks for Asynchronous Kafka Message Publishing

This article explains how to use Spring's TransactionSynchronizationManager to detect active transactions and register synchronization callbacks so that Kafka messages are sent asynchronously after transaction commit, illustrated with a payment‑system logging scenario and complete Java code examples.

AsynchronousBackendKafka
0 likes · 10 min read
Applying Spring Transaction Hooks for Asynchronous Kafka Message Publishing
37 Interactive Technology Team
37 Interactive Technology Team
Apr 15, 2024 · Backend Development

High-Performance Data Collection and Persistence Using OpenResty, Lua, and Kafka

The article describes a high‑throughput, fault‑tolerant data‑collection pipeline built with OpenResty, Lua, and Kafka that writes incoming records to disk using a cross‑worker mutex and cached file handles, rotates files via Nginx reload, and achieves over 13× performance versus typical online methods.

DataIngestionFileIOHighPerformance
0 likes · 7 min read
High-Performance Data Collection and Persistence Using OpenResty, Lua, and Kafka
Rare Earth Juejin Tech Community
Rare Earth Juejin Tech Community
Apr 6, 2024 · Big Data

Deep Dive into Kafka’s Underlying Mechanisms: Sequential Writes, Sparse Indexing, Segment Storage, and Replication

This article explores Apache Kafka’s core storage architecture, explaining how sequential append‑only writes, sparse indexing, segmented log files, and a leader‑based replication mechanism together enable high‑throughput, reliable, and scalable event streaming for massive data workloads.

Big DataEvent StreamingKafka
0 likes · 11 min read
Deep Dive into Kafka’s Underlying Mechanisms: Sequential Writes, Sparse Indexing, Segment Storage, and Replication
Sanyou's Java Diary
Sanyou's Java Diary
Apr 1, 2024 · Backend Development

7 Real-World Message Queue Patterns Every Backend Engineer Should Know

This article shares seven classic use cases of message queues—including asynchronous decoupling, traffic smoothing, bus architecture, delayed tasks, broadcast consumption, distributed transactions, and data hub integration—illustrated with real-world experiences and code examples to help engineers design robust high‑concurrency systems.

Broadcast ConsumptionKafkaRocketMQ
0 likes · 12 min read
7 Real-World Message Queue Patterns Every Backend Engineer Should Know
Architect
Architect
Mar 30, 2024 · Backend Development

11 Ways to Auto-Close Expired Orders: From Passive to Distributed Queues

The article systematically compares eleven technical approaches for automatically closing expired e‑commerce orders, detailing each method’s implementation steps, trade‑offs, performance characteristics, and ideal deployment scenarios, and finally ranks them to guide practical selection.

Backend ArchitectureDistributed SchedulingKafka
0 likes · 19 min read
11 Ways to Auto-Close Expired Orders: From Passive to Distributed Queues
Su San Talks Tech
Su San Talks Tech
Mar 30, 2024 · Backend Development

7 Real-World Message Queue Patterns Every Backend Engineer Should Know

From asynchronous decoupling to distributed transactions, this article explores seven classic message‑queue use cases—such as peak‑shaving, event buses, delayed tasks, broadcast consumption, and data hub integration—illustrated with real‑world examples from e‑commerce, ride‑hailing, and lottery systems, plus RocketMQ code snippets.

Distributed SystemsKafkaMessage Queue
0 likes · 13 min read
7 Real-World Message Queue Patterns Every Backend Engineer Should Know
HelloTech
HelloTech
Mar 21, 2024 · Big Data

Streaming Prediction System Construction and Real‑time Feature Templatization

The article describes how a Flink‑based streaming prediction platform was built to flatten peak request loads, reduce latency and memory use, and improve stability by deduplicating SDK calls, incrementally loading Hive features, partitioned caching, and comprehensive monitoring, while a templating system automates feature definition, SQL generation and stress testing, enabling real‑time supply‑demand forecasting that outperforms offline methods.

AIFlinkKafka
0 likes · 8 min read
Streaming Prediction System Construction and Real‑time Feature Templatization
Java Tech Enthusiast
Java Tech Enthusiast
Mar 16, 2024 · Backend Development

Common Causes of 100% CPU Usage in Java Backend Services

Typical reasons for Java backend services hitting 100 % CPU include fetching massive data sets, misusing Kafka auto‑acknowledge, infinite loops or recursion, thread‑pool overload causing context‑switch thrashing, bulk file synchronization, lock‑order deadlocks, poorly designed regular expressions, and intensive real‑time calculations under high concurrency.

CPUKafkajava
0 likes · 8 min read
Common Causes of 100% CPU Usage in Java Backend Services
IT Services Circle
IT Services Circle
Mar 9, 2024 · Backend Development

Java Backend Interview Guide: Networks, Data Structures, MySQL Logs, Elasticsearch, Kafka, Microservices, and Spring Essentials

This article compiles a comprehensive Java backend interview guide covering network protocols, HTTP characteristics, core data structures and algorithms, MySQL logging mechanisms, Elasticsearch full‑text search, Kafka reliability and deduplication, common micro‑service components, load‑balancing algorithms, and essential Spring concepts such as IoC, AOP, transactions, and MVC workflow.

BackendElasticsearchKafka
0 likes · 22 min read
Java Backend Interview Guide: Networks, Data Structures, MySQL Logs, Elasticsearch, Kafka, Microservices, and Spring Essentials
Architect's Guide
Architect's Guide
Mar 2, 2024 · Fundamentals

RabbitMQ vs Kafka: Core Differences and When to Use Each

This article compares RabbitMQ and Apache Kafka across architecture, message ordering, routing, timing, retention, fault handling, scalability, and consumer complexity, and provides guidance on which platform suits specific use‑cases such as flexible routing, strict ordering, long‑term retention, or high throughput.

KafkaMessage OrderingMessage Queue
0 likes · 19 min read
RabbitMQ vs Kafka: Core Differences and When to Use Each
Airbnb Technology Team
Airbnb Technology Team
Mar 1, 2024 · Big Data

Riverbed: A Scalable Data Framework for Real‑time and Batch Processing at Airbnb

Airbnb’s Riverbed framework unifies streaming CDC events and batch Spark jobs behind a GraphQL‑based declarative API to automatically build and maintain distributed materialized views, using Kafka‑partitioned ordering and version control to deliver billions of daily updates with low‑latency reads for features such as payments and search.

AirbnbApache SparkKafka
0 likes · 8 min read
Riverbed: A Scalable Data Framework for Real‑time and Batch Processing at Airbnb
IT Services Circle
IT Services Circle
Feb 27, 2024 · Backend Development

Common Causes of 100% CPU Usage in Java Backend Systems and Their Solutions

This article analyzes typical reasons why Java backend services experience sustained 100% CPU usage—such as bulk data retrieval, Kafka auto‑acknowledgement, infinite loops, excessive multithreading, massive file synchronization, deadlocks, inefficient regular expressions, and heavy real‑time calculations—and offers practical mitigation strategies.

CPUKafkadeadlock
0 likes · 9 min read
Common Causes of 100% CPU Usage in Java Backend Systems and Their Solutions
Su San Talks Tech
Su San Talks Tech
Feb 26, 2024 · Backend Development

Why Does My Java Service Hit 100% CPU? 8 Real-World Causes and Fixes

This article examines eight common reasons why Java backend services experience 100% CPU usage—such as fetching excessive data, automatic Kafka acknowledgments, infinite loops, multithreaded imports, massive file syncs, deadlocks, inefficient regular expressions, and heavy real‑time calculations—and provides practical mitigation strategies.

CPUKafkajava
0 likes · 9 min read
Why Does My Java Service Hit 100% CPU? 8 Real-World Causes and Fixes
MaGe Linux Operations
MaGe Linux Operations
Feb 20, 2024 · Big Data

Redis Streams vs Kafka: Which Is Better for Real‑Time Event Processing?

This article compares Redis Streams and Kafka, examining their architectures, ordering guarantees, consumer group models, scalability, and trade‑offs, and shows how Redis can emulate Kafka‑like semantics using the Runnel library, while highlighting memory‑speed benefits versus Kafka’s durable, unlimited log storage.

Event ProcessingKafkaRunnel
0 likes · 9 min read
Redis Streams vs Kafka: Which Is Better for Real‑Time Event Processing?
Alibaba Cloud Developer
Alibaba Cloud Developer
Feb 19, 2024 · Big Data

Master Kafka: Core Concepts, Architecture, and Interview Essentials

This comprehensive guide explains why Kafka is the industry standard for real‑time data pipelines, compares it with RabbitMQ and RocketMQ, details its architecture, APIs, leader election, offset management, idempotence, rebalance handling, and provides practical interview questions and code examples for engineers.

Kafka
0 likes · 19 min read
Master Kafka: Core Concepts, Architecture, and Interview Essentials
Programmer DD
Programmer DD
Feb 17, 2024 · Backend Development

How to Seamlessly Upgrade Spring Boot 2.x to 3.x: Step‑by‑Step Guide

This guide explains why upgrading from Spring Boot 2.x to 3.x is essential, outlines the required JDK 17 migration, Spring Boot version bump, configuration property changes, Jakarta EE transition, and updates to Spring Security, Kafka, RestTemplate, and OpenAPI integration, providing code examples for each step.

KafkaOpenAPISpring Boot
0 likes · 10 min read
How to Seamlessly Upgrade Spring Boot 2.x to 3.x: Step‑by‑Step Guide
Sanyou's Java Diary
Sanyou's Java Diary
Feb 1, 2024 · Backend Development

Message Queues Unveiled: From Decoupling to Platformization and Core Architectures

This article traces the two‑decade evolution of message queues—from early decoupling solutions like ActiveMQ, through high‑throughput designs such as Kafka, to modern platformized systems like RocketMQ and Pulsar—while explaining fundamental concepts, partitioning, and storage architectures that underpin today’s distributed messaging platforms.

Distributed SystemsKafkaMessage Queue
0 likes · 17 min read
Message Queues Unveiled: From Decoupling to Platformization and Core Architectures
Efficient Ops
Efficient Ops
Jan 31, 2024 · Databases

Why ClickHouse Beats Elasticsearch for High‑Performance Log Analytics

Facing data security and cost challenges in SaaS, the author evaluates ClickHouse versus Elasticsearch, highlighting ClickHouse’s superior write throughput, query speed, lower storage and CPU usage, and provides detailed deployment guides for Zookeeper, Kafka, FileBeat, and ClickHouse to build a cost‑effective private analytics platform.

Big DataDatabase DeploymentElasticsearch
0 likes · 8 min read
Why ClickHouse Beats Elasticsearch for High‑Performance Log Analytics
Mike Chen's Internet Architecture
Mike Chen's Internet Architecture
Jan 27, 2024 · Backend Development

Understanding Message Queues: Concepts, Use Cases, Types, and Common Implementations

This article introduces message queues, explaining their role in high‑traffic systems, key characteristics such as asynchronous communication, decoupling, reliability and buffering, compares point‑to‑point and publish‑subscribe models, and reviews popular implementations like RabbitMQ, Kafka, ActiveMQ, RocketMQ and Pulsar.

DecouplingKafkaMessage Queue
0 likes · 8 min read
Understanding Message Queues: Concepts, Use Cases, Types, and Common Implementations
MaGe Linux Operations
MaGe Linux Operations
Jan 21, 2024 · Big Data

Master Kafka: Core Concepts, Metrics, and Troubleshooting Guide

This article explains Kafka's fundamental components, version evolution, key monitoring metrics for producers, brokers, consumers and Zookeeper, and provides step‑by‑step troubleshooting methods for common issues such as slow topic throughput and message backlog.

Big DataKafkaMessage Queue
0 likes · 8 min read
Master Kafka: Core Concepts, Metrics, and Troubleshooting Guide
Sanyou's Java Diary
Sanyou's Java Diary
Jan 18, 2024 · Big Data

Inside Kafka’s Network Layer: How Messages Are Sent and Received

This article walks through Kafka’s network‑layer internals, detailing how the client prepares, pre‑sends, and actually sends messages, then reads and parses responses, with code snippets that illustrate the Selector, KafkaChannel, and related classes in Java NIO.

Java NIOKafkaMessage Receiving
0 likes · 22 min read
Inside Kafka’s Network Layer: How Messages Are Sent and Received