BestHub
Sign in
Tagged articles
1273 articles
Page 1 of 13
Su San Talks Tech
Su San Talks Tech
May 19, 2026 · Interview Experience

Designing a Hundred‑Billion‑Scale Message Queue: A ByteDance Interview Walkthrough

This article walks through the interview question of designing a message queue that handles billions of messages daily and peaks at millions of QPS, covering traffic calculations, core roles, storage and throughput techniques, scalability, high availability, observability, framework comparisons, a real‑world case study, and key follow‑up interview topics.

High ThroughputKafkaMessage Queue
0 likes · 12 min read
Designing a Hundred‑Billion‑Scale Message Queue: A ByteDance Interview Walkthrough
Lobster Programming
Lobster Programming
May 6, 2026 · Backend Development

How to Choose the Right MQ: RabbitMQ vs RocketMQ vs Kafka

This article compares RabbitMQ, RocketMQ, and Kafka on throughput, latency, scalability, and reliability, outlining each system's core features and recommending suitable scenarios such as reliable messaging, high‑performance streaming, and large‑scale real‑time data processing.

KafkaLatencyMessage Queue
0 likes · 6 min read
How to Choose the Right MQ: RabbitMQ vs RocketMQ vs Kafka
Woodpecker Software Testing
Woodpecker Software Testing
Apr 30, 2026 · Databases

Datafaker: A Powerful Tool for Bulk Test Data Generation

Datafaker is a Python‑compatible utility that creates large volumes of synthetic test data for databases, streams, files, and messaging systems, offering flexible metadata rules, multi‑backend support, and command‑line options for quick data provisioning.

ElasticsearchKafkaPython
0 likes · 14 min read
Datafaker: A Powerful Tool for Bulk Test Data Generation
DevOps Coach
DevOps Coach
Apr 26, 2026 · Backend Development

Forget Kafka: A Lightweight Go Queue Achieves 2 Million Messages per Second

The article analyzes how replacing Kafka with a simple in‑memory Go queue reduced architectural complexity, boosted throughput from 240‑330 K to 1.8‑2.0 M messages per second, and clarified debugging, while still acknowledging scenarios where Kafka remains the better choice.

Backend PerformanceGoIn‑Memory Ring Buffer
0 likes · 8 min read
Forget Kafka: A Lightweight Go Queue Achieves 2 Million Messages per Second
ITPUB
ITPUB
Apr 17, 2026 · Industry Insights

Why LinkedIn Dumped Kafka for Its Own ‘Northguard’ Streaming Engine

LinkedIn, the original home of Apache Kafka, abandoned the platform for a home‑grown system called Northguard, redesigning log storage, decentralizing metadata, and adding a virtualized Xinfra layer to handle trillions of daily events, while still acknowledging Kafka’s relevance for most companies.

Distributed SystemsInfrastructureKafka
0 likes · 7 min read
Why LinkedIn Dumped Kafka for Its Own ‘Northguard’ Streaming Engine
Architect Chen
Architect Chen
Apr 16, 2026 · Big Data

Supercharge Kafka Consumer Performance: Parallelism, Batching, and Multithreading

This guide explains practical techniques to dramatically increase Kafka consumer throughput, including scaling consumer instances or partitions, tuning fetch and poll parameters, and implementing a multithreaded consumer model, while also covering hardware, JVM, and OS optimizations and monitoring recommendations.

Batch FetchConsumer ParallelismKafka
0 likes · 5 min read
Supercharge Kafka Consumer Performance: Parallelism, Batching, and Multithreading
Ray's Galactic Tech
Ray's Galactic Tech
Apr 4, 2026 · Backend Development

How to Build a High‑Concurrency Story Creation Platform with AgentScope Java

This article presents a step‑by‑step engineering guide for constructing a production‑grade, high‑throughput story generation platform using AgentScope Java, Spring Boot, Kafka, Redis, PostgreSQL, and Kubernetes, covering architecture, task modeling, DAG orchestration, code organization, scalability, observability, and deployment best practices.

JavaKafkaMulti-Agent
0 likes · 39 min read
How to Build a High‑Concurrency Story Creation Platform with AgentScope Java
Alibaba Cloud Native
Alibaba Cloud Native
Mar 30, 2026 · Industry Insights

How Haier’s AIoT Platform Scaled to Billions of Messages with Kafka Serverless on Alibaba Cloud

The article details how Haier Smart Home’s AIoT platform tackled massive device messaging demands by migrating its self‑built Kafka clusters to Alibaba Cloud’s Kafka Serverless, outlining the technical challenges, step‑by‑step migration plan, custom performance tuning, risk‑co‑governance, and the resulting improvements in stability, throughput, and operational efficiency.

AIoTAlibaba CloudKafka
0 likes · 11 min read
How Haier’s AIoT Platform Scaled to Billions of Messages with Kafka Serverless on Alibaba Cloud
360 Zhihui Cloud Developer
360 Zhihui Cloud Developer
Mar 27, 2026 · Cloud Native

How AutoMQ Transforms Kafka into a Cloud‑Native, Elastic Messaging Service

This article examines the limitations of traditional Kafka in large‑scale deployments and presents AutoMQ’s cloud‑native redesign—detailing its stateless architecture, storage separation, automatic scaling, read/write isolation, performance benchmarks, and real‑world migration case studies that demonstrate reduced latency, higher throughput, and lower resource costs.

AutoMQCloud NativeKafka
0 likes · 13 min read
How AutoMQ Transforms Kafka into a Cloud‑Native, Elastic Messaging Service
Alibaba Cloud Native
Alibaba Cloud Native
Mar 20, 2026 · Cloud Native

How a Gaming Platform Scaled to Millions with RocketMQ & Kafka: A Cloud‑Native Success Story

Facing explosive growth, the game‑service platform 悠悠有品 rebuilt its architecture on Alibaba Cloud, using RocketMQ for core transaction messaging and Kafka for data synchronization, achieving elastic scaling, high availability, cost reduction, and reliable high‑concurrency processing across its trading and analytics pipelines.

KafkaMessagingRocketMQ
0 likes · 8 min read
How a Gaming Platform Scaled to Millions with RocketMQ & Kafka: A Cloud‑Native Success Story
dbaplus Community
dbaplus Community
Mar 5, 2026 · Backend Development

How to Ensure Message Order in Kafka: From Basics to Advanced Solutions

This article explains the concept of message ordering in distributed systems, details how Kafka stores messages in partitions, compares global and partial ordering, evaluates single‑partition, asynchronous, and multi‑partition solutions—including handling data skew and partition expansion—and provides a practical interview guide.

BackendDistributed SystemsKafka
0 likes · 22 min read
How to Ensure Message Order in Kafka: From Basics to Advanced Solutions
Architect Chen
Architect Chen
Mar 3, 2026 · Backend Development

Preventing Kafka Duplicate Consumption with Idempotent Design

This article explains practical strategies to avoid duplicate message consumption in Kafka, covering business idempotency with unique IDs, database or Redis deduplication tables, enabling producer idempotence, consumer-side checks, and Kafka's transaction-based exactly‑once semantics, along with their trade‑offs and suitable scenarios.

BackendExactly-OnceIdempotency
0 likes · 4 min read
Preventing Kafka Duplicate Consumption with Idempotent Design
ITPUB
ITPUB
Feb 11, 2026 · Backend Development

How to Guarantee Zero Message Loss in MQ Systems: A Full‑Lifecycle Design

This guide explains why guaranteeing 100% message reliability in MQ is a critical system‑design interview topic and presents a three‑layer architecture—production, storage, and consumption—detailing ACK settings, local message tables, broker replication, leader election safeguards, manual offset commits, and idempotent processing to prevent any message loss.

AcknowledgmentDistributed SystemsIdempotency
0 likes · 11 min read
How to Guarantee Zero Message Loss in MQ Systems: A Full‑Lifecycle Design
ITPUB
ITPUB
Feb 9, 2026 · Databases

ClickHouse vs Doris vs Redis: Real‑World Query Performance Test with Flink

Using a 600k‑record IP range dataset, we built identical tables in ClickHouse and Doris, and a Redis skip‑list store, then ran three Flink‑Kafka streaming jobs to compare query latency across the three databases under varying traffic rates, revealing Redis as fastest, ClickHouse second, Doris slowest.

ClickHouseDatabase PerformanceFlink
0 likes · 8 min read
ClickHouse vs Doris vs Redis: Real‑World Query Performance Test with Flink
Top Architect
Top Architect
Feb 4, 2026 · Backend Development

Build a Robust Asynchronous Processing SDK with Spring, Kafka and MySQL

This article introduces a generic asynchronous processing SDK for Java back‑ends, explaining its design principles, advantages, component architecture, database schema, configuration via Apollo, usage steps, and practical demonstrations, while providing complete code snippets and a GitHub repository for reference.

AsynchronousJavaKafka
0 likes · 12 min read
Build a Robust Asynchronous Processing SDK with Spring, Kafka and MySQL
Tencent Cloud Developer
Tencent Cloud Developer
Feb 4, 2026 · Backend Development

How We Cut Server Costs by 82%: Refactoring a High‑Concurrency QQ Game Service from C++ to Go with Kafka

This article details the redesign of a core QQ game achievement service that suffered from low resource utilization and heavy CAS contention, describing how moving from a synchronous C++ implementation to an asynchronous Go‑Kafka pipeline eliminated lock conflicts, reduced server count by 82%, and dramatically improved latency and stability.

Kafkahigh concurrencyrefactoring
0 likes · 11 min read
How We Cut Server Costs by 82%: Refactoring a High‑Concurrency QQ Game Service from C++ to Go with Kafka
Java Tech Enthusiast
Java Tech Enthusiast
Feb 3, 2026 · Backend Development

Spring Boot 4.0.2: Critical Kafka Fixes, Dependency Upgrades & What to Watch

Spring Boot 4.0.2 is a maintenance release that focuses on fixing over 20 bugs—including a critical Kafka transaction auto‑configuration issue—upgrading more than 40 core dependencies, improving documentation, and introducing minor breaking changes, making it essential for Java developers to upgrade promptly.

Kafkabug fixdependency-upgrade
0 likes · 8 min read
Spring Boot 4.0.2: Critical Kafka Fixes, Dependency Upgrades & What to Watch
Ray's Galactic Tech
Ray's Galactic Tech
Jan 30, 2026 · Cloud Native

Scale a Monolithic Article Interaction Service with Kubernetes Microservices

This article walks through converting a single‑service article interaction module—handling likes, favorites, and reads—into independent microservices deployed on Kubernetes, detailing architecture goals, service separation, Redis‑based high‑concurrency handling, Kafka async persistence, deployment configurations, auto‑scaling, and real‑world performance results.

Kafkaredis
0 likes · 8 min read
Scale a Monolithic Article Interaction Service with Kubernetes Microservices
Su San Talks Tech
Su San Talks Tech
Jan 30, 2026 · Backend Development

Why Kafka Rebalance Causes Backlog, Duplicates, and Data Loss—and How to Fix It

Kafka consumer group rebalances can trigger message backlogs, duplicate processing, and data loss; this article explains common rebalance triggers, their impact on consumption, and practical configuration and coding strategies—such as tuning timeout parameters, using manual offset commits, and sticky partition assignment—to minimize disruptions.

Data lossKafkaMessage Duplication
0 likes · 12 min read
Why Kafka Rebalance Causes Backlog, Duplicates, and Data Loss—and How to Fix It
DevOps Coach
DevOps Coach
Jan 27, 2026 · Backend Development

7 Essential Kafka Design Patterns Every Engineer Should Master

This guide presents seven practical Kafka design patterns—single‑key single‑write, log compaction, multi‑consumer‑group fan‑out, retry and dead‑letter topics, exactly‑once processing with Streams, schema evolution with Avro, and choreography vs orchestration—detailing when to use each, core principles, code examples, tips, common pitfalls, and final recommendations for building reliable, observable, and maintainable event‑driven systems.

Design PatternsEvent StreamingKafka
0 likes · 9 min read
7 Essential Kafka Design Patterns Every Engineer Should Master
Ray's Galactic Tech
Ray's Galactic Tech
Jan 23, 2026 · Backend Development

How to Build a Kafka‑Level High‑Performance Message Queue from Scratch

This article presents a step‑by‑step guide to designing and implementing a Kafka‑class distributed log‑based message queue kernel, covering architecture, sequential writes, sparse indexing, zero‑copy I/O, partitioning, replication, consumer‑group metadata, batch pipelines, crash recovery, and performance benchmarks.

KafkaMessage QueueReplication
0 likes · 7 min read
How to Build a Kafka‑Level High‑Performance Message Queue from Scratch
Architect's Guide
Architect's Guide
Jan 22, 2026 · Big Data

Unlock Kafka’s Power: Core Concepts, High‑Performance Architecture & Real‑World Scaling Tips

This comprehensive guide explores Kafka’s core value as a message queue, explains producers, consumers, topics, partitions, and replication, dives into cluster architecture, zero‑copy I/O, resource planning for disks, memory, CPU and network, and provides practical configuration, consumer‑group management, and operational tooling tips for building high‑throughput, highly available Kafka deployments.

Distributed SystemsKafkaMessage Queue
0 likes · 31 min read
Unlock Kafka’s Power: Core Concepts, High‑Performance Architecture & Real‑World Scaling Tips
ITPUB
ITPUB
Jan 21, 2026 · Interview Experience

How to Design a Billion‑User Real‑Time Step Leaderboard for Interviews

This article breaks down the interview‑level system design of a WeChat‑style step leaderboard that must support over a billion users, handling massive write spikes, low‑latency friend ranking queries, storage scaling, and relationship complexity with a three‑part architecture using MQ, Redis, and MySQL.

KafkaSystem Designhigh concurrency
0 likes · 8 min read
How to Design a Billion‑User Real‑Time Step Leaderboard for Interviews
Tech Freedom Circle
Tech Freedom Circle
Jan 15, 2026 · Backend Development

Kafka Rebalance Storm Crushed 120k QPS in JD Interview – How to Understand and Fix

In a JD senior Java architect interview, a Kafka consumer‑group rebalance storm caused QPS to drop from 120k to zero, triggering massive message loss and latency spikes, and the article walks through the rebalance fundamentals, failure causes, impact analysis, cooperative sticky assignor migration, and comprehensive monitoring and mitigation strategies.

Distributed SystemsKafkaconsumer-group
0 likes · 28 min read
Kafka Rebalance Storm Crushed 120k QPS in JD Interview – How to Understand and Fix
Code Wrench
Code Wrench
Jan 14, 2026 · Backend Development

When to Choose NATS Over Kafka for Go Microservices: A Practical Guide

This article compares Kafka, RabbitMQ, and NATS for Go microservices, explains why Kafka is often over‑engineered for internal communication, and shows how NATS provides a lightweight, event‑driven alternative with concrete code examples and a clear selection matrix.

GoKafkaMessage Queue
0 likes · 9 min read
When to Choose NATS Over Kafka for Go Microservices: A Practical Guide
JD Retail Technology
JD Retail Technology
Jan 13, 2026 · Backend Development

Deep Dive into Kafka, RocketMQ, and JMQ Storage Architectures

This article compares the storage models, data organization, indexing, read/write processes, and performance trade‑offs of three major message queues—Kafka, RocketMQ, and JMQ—providing detailed technical insights for architects and engineers making storage‑related design decisions.

Backend EngineeringJMQKafka
0 likes · 16 min read
Deep Dive into Kafka, RocketMQ, and JMQ Storage Architectures
Top Architect
Top Architect
Jan 12, 2026 · Backend Development

How to Build a Robust Asynchronous Processing SDK with Spring, Kafka, and XXL‑Job

This article explains the design and implementation of a generic asynchronous processing SDK for Java, covering its purpose, advantages, core principles, component choices, design patterns, configuration via Apollo, usage steps, safety considerations, and provides complete SQL and Spring configuration examples along with a GitHub repository link.

AsynchronousDesign PatternsJava
0 likes · 11 min read
How to Build a Robust Asynchronous Processing SDK with Spring, Kafka, and XXL‑Job
iQIYI Technical Product Team
iQIYI Technical Product Team
Jan 8, 2026 · Big Data

How iQIYI Cut Stream Data Costs by 70%: From Private‑Cloud Kafka to AutoMQ

This article details iQIYI's evolution from a tightly coupled private‑cloud Kafka setup to a cloud‑native AutoMQ architecture, describing the challenges of scaling, the development of the Stream platform and Stream‑SDK, the migration to hybrid and public‑cloud Kafka, and the resulting cost and elasticity improvements.

AutoMQData ArchitectureKafka
0 likes · 12 min read
How iQIYI Cut Stream Data Costs by 70%: From Private‑Cloud Kafka to AutoMQ
Ray's Galactic Tech
Ray's Galactic Tech
Jan 6, 2026 · Backend Development

Scalable GPS Data Backend: SpringBoot, Kafka, MongoDB & Redis Design

This guide outlines a complete backend architecture for high‑volume GPS data, detailing the overall system flow, technology stack choices, Maven dependencies, data models, Kafka producer/consumer configurations, SpringBoot controllers, asynchronous processing, Redis caching, health checks, Docker deployment, and performance tuning recommendations to ensure stability and scalability.

DockerKafkaMongoDB
0 likes · 11 min read
Scalable GPS Data Backend: SpringBoot, Kafka, MongoDB & Redis Design
Tech Freedom Circle
Tech Freedom Circle
Jan 6, 2026 · Backend Development

Why Choose RocketMQ Over Kafka? The Real Reasons Behind the 90% Mistake

This article dissects a common interview question about Kafka's higher throughput versus RocketMQ's richer features, explains the underlying design philosophies, storage models, I/O paths, scaling limits, real‑world use cases such as transaction, delayed and ordered messages, and provides concrete optimization steps and code samples to help engineers make an informed messaging platform choice.

Distributed SystemsJavaKafka
0 likes · 42 min read
Why Choose RocketMQ Over Kafka? The Real Reasons Behind the 90% Mistake
JakartaEE China Community
JakartaEE China Community
Jan 6, 2026 · Backend Development

Run a WildFly Application with JBang in Minutes

This tutorial shows how to use JBang scripts to launch a WildFly server, create a simple Jakarta REST endpoint, and extend the server with WildFly Glow add‑ons such as Kafka, providing step‑by‑step commands, code examples, and Docker integration for rapid prototyping.

JBangJava scriptingKafka
0 likes · 9 min read
Run a WildFly Application with JBang in Minutes
Architect Chen
Architect Chen
Jan 2, 2026 · Backend Development

Preventing Duplicate Consumption in Kafka: Design, Idempotence, and Configuration Strategies

This guide explains how to avoid duplicate message consumption in Kafka by designing unique identifiers, implementing consumer-side idempotence with deduplication tables, leveraging Kafka’s transactional features, and establishing system-level safeguards and monitoring to ensure reliable, exactly‑once processing.

Backend DevelopmentDuplicate ConsumptionExactly-Once
0 likes · 4 min read
Preventing Duplicate Consumption in Kafka: Design, Idempotence, and Configuration Strategies
dbaplus Community
dbaplus Community
Dec 30, 2025 · Backend Development

How to Tackle Massive Message Queue Backlogs in High‑Traffic Scenarios

During peak traffic like Double‑11, a message queue can accumulate millions of messages, and simply adding consumer instances only offers temporary relief; this article explains the partition model limits, how to calculate proper partition numbers, fast remediation tactics, and deep consumer‑side optimizations for robust, scalable processing.

BacklogKafkaMessage Queue
0 likes · 20 min read
How to Tackle Massive Message Queue Backlogs in High‑Traffic Scenarios
Mike Chen's Internet Architecture
Mike Chen's Internet Architecture
Dec 25, 2025 · Backend Development

How to Resolve Kafka Backlog Under High Load: Practical Tips

This article explains why Kafka experiences message backlog in high‑load environments, identifies producer‑consumer speed mismatches, I/O and resource bottlenecks, and offers concrete strategies such as scaling consumers, tuning hardware, and adjusting Kafka configurations to eliminate the backlog.

BacklogDistributed SystemsKafka
0 likes · 4 min read
How to Resolve Kafka Backlog Under High Load: Practical Tips
Mike Chen's Internet Architecture
Mike Chen's Internet Architecture
Dec 22, 2025 · Operations

How to Diagnose and Resolve Kafka Consumer Lag Quickly

When Kafka consumers fall behind, this guide walks you through confirming the backlog, pinpointing bottlenecks in production, consumption, or brokers, and applying concrete steps—such as checking offsets, comparing TPS, inspecting consumer logic, and adjusting partitions—to efficiently eliminate lag.

Consumer LagKafka
0 likes · 4 min read
How to Diagnose and Resolve Kafka Consumer Lag Quickly
Architect Chen
Architect Chen
Dec 21, 2025 · Backend Development

How to Resolve Kafka Backlog: Boost Consumer Throughput and Optimize Partitions

This guide explains why Kafka backlog occurs when production outpaces consumption and provides practical steps—such as increasing consumer instances, optimizing processing, expanding partitions, applying flow‑control, and managing message capacity—to eliminate the backlog and keep the cluster healthy.

BacklogFlow ControlKafka
0 likes · 4 min read
How to Resolve Kafka Backlog: Boost Consumer Throughput and Optimize Partitions
Mike Chen's Internet Architecture
Mike Chen's Internet Architecture
Dec 9, 2025 · Backend Development

Boost Kafka to Over 1 Million Messages per Second: Metrics and Tuning Tips

This article explains what high concurrency means for Kafka, outlines key performance metrics such as QPS, TPS, throughput and latency, and provides concrete configuration and architectural techniques—including broker optimization, horizontal scaling, network batching, and zero‑copy—to achieve write rates exceeding one million records per second.

BackendDistributed SystemsKafka
0 likes · 4 min read
Boost Kafka to Over 1 Million Messages per Second: Metrics and Tuning Tips
IT Services Circle
IT Services Circle
Dec 9, 2025 · Backend Development

Mastering Kafka Rebalance: Prevent Backlog, Duplicates, and Data Loss

When Kafka consumer groups rebalance, partitions are reassigned, often causing message backlog, duplicate processing, or loss; understanding the triggers, impact, and optimization techniques—like tuning timeouts, managing offset commits, and using sticky assignors—can keep your streaming pipelines reliable.

Duplicate ConsumptionKafkaOffset Management
0 likes · 13 min read
Mastering Kafka Rebalance: Prevent Backlog, Duplicates, and Data Loss
dbaplus Community
dbaplus Community
Dec 8, 2025 · Databases

Which Database Wins IP Range Lookups? ClickHouse vs Doris vs Redis Benchmarks

This article presents a systematic benchmark comparing ClickHouse, Doris, and Redis for IP‑range dimension lookups using Flink‑Kafka pipelines, detailing test design, result table schema, query interfaces, and performance results across varying data rates, concluding that Redis offers the fastest and most stable query latency.

ClickHouseDatabase BenchmarkFlink
0 likes · 7 min read
Which Database Wins IP Range Lookups? ClickHouse vs Doris vs Redis Benchmarks
Mike Chen's Internet Architecture
Mike Chen's Internet Architecture
Dec 3, 2025 · Big Data

Kafka High‑Throughput Tricks: Sequential Writes, Zero‑Copy, Partitioning

The article explains how Kafka achieves high throughput by writing messages sequentially to disk, leveraging OS page cache and zero‑copy system calls, using partitioned topics for parallelism, batching and compressing records on both producer and broker sides, and employing asynchronous replication with configurable persistence strategies.

BatchingHigh ThroughputKafka
0 likes · 4 min read
Kafka High‑Throughput Tricks: Sequential Writes, Zero‑Copy, Partitioning
Code Wrench
Code Wrench
Nov 22, 2025 · Backend Development

Build a Production-Ready Rule Engine with Gray Release Using Go, Kafka, and Redis

Learn how to design and implement a ready-to-use rule engine combined with a gray release system using Golang, Kafka, Redis, and CEL, complete with Docker‑compose deployment, edge execution, token‑bucket throttling, and webhook actions, plus full source code for a production‑grade marketing strategy platform.

CELGoKafka
0 likes · 9 min read
Build a Production-Ready Rule Engine with Gray Release Using Go, Kafka, and Redis
dbaplus Community
dbaplus Community
Nov 18, 2025 · Backend Development

How to Guarantee 100% No Message Loss in Distributed MQ Systems

Ensuring that messages never disappear in a distributed MQ system requires a three‑pronged strategy covering production, storage, and consumption, with proper ACK configurations, local message tables, replication settings, and manual offset commits to achieve reliable, at‑least‑once processing without data loss.

BackendKafkaMQ
0 likes · 11 min read
How to Guarantee 100% No Message Loss in Distributed MQ Systems
MaGe Linux Operations
MaGe Linux Operations
Nov 18, 2025 · Big Data

Zero‑Data‑Loss Kafka Cluster Scaling: Complete Step‑by‑Step Guide

This comprehensive guide explains how to safely expand a Kafka cluster without data loss by covering applicable scenarios, pre‑conditions, anti‑pattern warnings, environment matrices, a detailed checklist, step‑by‑step Linux commands for broker preparation, partition‑rebalancing plan generation, throttled execution, real‑time monitoring, verification, rollback procedures, backup strategies, performance testing, common troubleshooting, FAQs and best‑practice scripts, all illustrated with code snippets and practical examples.

KafkaLinuxPartition Rebalancing
0 likes · 47 min read
Zero‑Data‑Loss Kafka Cluster Scaling: Complete Step‑by‑Step Guide
Ray's Galactic Tech
Ray's Galactic Tech
Nov 17, 2025 · Backend Development

Kafka vs RabbitMQ vs RocketMQ: Which Message Broker Fits Your Use Case?

This article provides an in‑depth, multi‑dimensional comparison of Apache Kafka, RabbitMQ, and Apache RocketMQ—covering design philosophy, performance, reliability, features, ecosystem, operations, and typical scenarios—to help you choose the right message middleware for your architecture.

KafkaRabbitMQRocketMQ
0 likes · 6 min read
Kafka vs RabbitMQ vs RocketMQ: Which Message Broker Fits Your Use Case?
Tech Freedom Circle
Tech Freedom Circle
Nov 15, 2025 · Databases

How to Prevent Order Loss in a 100k TPS Flash Sale When the Master DB Crashes – 5 Practical Solutions

The article dissects a high‑traffic flash‑sale interview question—how to guarantee zero order loss at 100,000 TPS when the master MySQL instance fails—by explaining the underlying performance‑consistency conflict, the three skills interviewers assess, and presenting five concrete, code‑driven solutions ranging from MySQL parameter tuning to semi‑sync replication, local message tables, group replication, and Redis‑Kafka traffic shaping.

Data ConsistencyGroup ReplicationKafka
0 likes · 28 min read
How to Prevent Order Loss in a 100k TPS Flash Sale When the Master DB Crashes – 5 Practical Solutions
mikechen
mikechen
Nov 13, 2025 · Backend Development

How to Diagnose and Resolve Kafka Message Backlog Issues

This article explains what Kafka message backlog is, outlines the main reasons it occurs—such as producer speed outpacing consumers, slow consumer processing, and downstream bottlenecks—and provides practical steps for producer throttling, consumer scaling and logic improvements, and Kafka cluster enhancements to eliminate the backlog.

Backend DevelopmentKafkamessage backlog
0 likes · 6 min read
How to Diagnose and Resolve Kafka Message Backlog Issues
Ray's Galactic Tech
Ray's Galactic Tech
Nov 9, 2025 · Backend Development

Hybrid Push‑Pull Timeline Architecture: Scaling Social Feeds for Billions

To serve billions of users with real‑time timelines, modern social platforms combine push‑based delivery for regular users and pull‑based retrieval for high‑profile accounts, employing hot‑cold separation, Kafka fan‑out, Redis caching, and scalable storage strategies to balance write and read loads.

Kafkapush-pullredis
0 likes · 9 min read
Hybrid Push‑Pull Timeline Architecture: Scaling Social Feeds for Billions
Ops Community
Ops Community
Nov 6, 2025 · Big Data

Zero Data Loss Kafka Cluster Scaling: From 3 to 10 Nodes – A Complete Guide

This comprehensive guide walks you through expanding or shrinking a production‑grade Kafka cluster—covering prerequisites, anti‑pattern warnings, environment matrices, step‑by‑step expansion and contraction procedures, partition rebalancing principles, monitoring, best practices, and troubleshooting—to ensure zero data loss during scaling.

Big DataKafkaPartition Rebalancing
0 likes · 27 min read
Zero Data Loss Kafka Cluster Scaling: From 3 to 10 Nodes – A Complete Guide
DataFunTalk
DataFunTalk
Nov 6, 2025 · Cloud Native

How Tencent Music Cut Kafka Costs by 50% with Cloud‑Native AutoMQ

Tencent Music migrated its massive Kafka streaming infrastructure to the cloud‑native AutoMQ platform, slashing operational costs by over half, achieving second‑level partition migration, and dramatically improving scaling efficiency while maintaining high‑throughput, low‑latency data processing for its music services.

AutoMQCost OptimizationData Streaming
0 likes · 16 min read
How Tencent Music Cut Kafka Costs by 50% with Cloud‑Native AutoMQ
Top Architect
Top Architect
Oct 31, 2025 · Backend Development

Mastering Message Queues: A Deep Dive into RabbitMQ, RocketMQ, and Kafka

This comprehensive guide explains the core components, exchange types, TTL, confirm mechanisms, consumer ACK/NACK, dead‑letter queues, and high‑availability features of RabbitMQ, RocketMQ, and Kafka, while also covering load balancing, ordering, transaction handling, and best practices for reliable message delivery.

Backend DevelopmentDistributed SystemsKafka
0 likes · 32 min read
Mastering Message Queues: A Deep Dive into RabbitMQ, RocketMQ, and Kafka
Big Data Technology & Architecture
Big Data Technology & Architecture
Oct 30, 2025 · Backend Development

What’s New in Apache Kafka 4.1? Core Features and Architecture Changes Explained

Apache Kafka 4.1.0 introduces native queue semantics, a new Streams rebalancing protocol, multi‑version Connect plugins, a revamped consumer‑group protocol, enhanced transaction safety, and numerous client, monitoring, and security improvements, offering a comprehensive upgrade over the 4.0 release.

KafkaStreamingdistributed-systems
0 likes · 6 min read
What’s New in Apache Kafka 4.1? Core Features and Architecture Changes Explained
Su San Talks Tech
Su San Talks Tech
Oct 28, 2025 · Backend Development

How to Prevent MQ Message Loss: 5 Proven Strategies for Reliable Messaging

Discover the three stages where MQ messages can be lost and explore five practical solutions—including producer confirmations, message persistence, consumer acknowledgments, transactional messaging, and retry with dead‑letter queues—complete with code examples and guidance on selecting the right approach for different scenarios.

Dead Letter QueueKafkaMessage Queue
0 likes · 14 min read
How to Prevent MQ Message Loss: 5 Proven Strategies for Reliable Messaging
Huolala Tech
Huolala Tech
Oct 22, 2025 · Backend Development

Scaling Real‑Time Reconciliation with Dynamic Kafka Consumer Clusters

To ensure fund safety and robust operations, the team built a real‑time reconciliation platform that leverages Kafka, and after encountering scaling bottlenecks with a static consumer model, they implemented a dynamic, partition‑level, weighted load‑balancing consumer cluster that supports automatic scaling and high‑throughput processing.

Backend ArchitectureDistributed SystemsDynamic Scaling
0 likes · 15 min read
Scaling Real‑Time Reconciliation with Dynamic Kafka Consumer Clusters
Architect Chen
Architect Chen
Oct 22, 2025 · Big Data

How to Eliminate Kafka Message Backlog with Practical Optimizations

This guide presents concrete techniques for improving Kafka consumer and producer performance, scaling clusters, tuning broker settings, and designing asynchronous buffering layers to prevent message accumulation and boost overall throughput.

Big DataKafkaPerformance Optimization
0 likes · 5 min read
How to Eliminate Kafka Message Backlog with Practical Optimizations
Raymond Ops
Raymond Ops
Oct 21, 2025 · Big Data

Deep Dive into Kafka Architecture: Topics, Partitions, and Reliable Data Pipelines

This article explains Kafka’s core concepts—including topics, partitions, log segmentation, indexing, and acknowledgment mechanisms—then provides a step‑by‑step guide to deploy a Zookeeper‑Kafka cluster integrated with Filebeat, Logstash, and the ELK stack for reliable log collection and analysis.

Big DataELKFilebeat
0 likes · 11 min read
Deep Dive into Kafka Architecture: Topics, Partitions, and Reliable Data Pipelines
Code Wrench
Code Wrench
Oct 18, 2025 · Backend Development

Master High‑Performance Queues in Go: Kafka, RabbitMQ & Redis Compared

This article explains how to build a high‑throughput, low‑latency, and scalable queue system in Go by leveraging Kafka, RabbitMQ, and Redis, covering core concepts, practical code examples, performance optimizations, and guidance on choosing the right solution for different workloads.

GoKafkaMessage Queue
0 likes · 11 min read
Master High‑Performance Queues in Go: Kafka, RabbitMQ & Redis Compared
StarRocks
StarRocks
Oct 14, 2025 · Big Data

How Ctrip Scaled UBT Analytics by Migrating from ClickHouse to StarRocks

Ctrip's User Behavior Tracking (UBT) system, handling 30 TB of daily data, moved from ClickHouse to StarRocks' compute‑storage separated architecture, cutting average query latency from 1.4 seconds to 203 ms, halving storage, reducing nodes from 50 to 40, and boosting write throughput to 3 million rows per second.

Big DataClickHouseData Migration
0 likes · 15 min read
How Ctrip Scaled UBT Analytics by Migrating from ClickHouse to StarRocks
Efficient Ops
Efficient Ops
Oct 12, 2025 · Information Security

What Exposed the AI Companion Apps? A Deep Dive into the Massive Data Leak

A recent breach of the AI companion apps Chattee and GiMe Chat exposed over 40 million private conversations, hundreds of thousands of media files, and transaction records due to publicly accessible Kafka brokers lacking any authentication, putting user privacy and security at serious risk.

AI securityKafkadata breach
0 likes · 3 min read
What Exposed the AI Companion Apps? A Deep Dive into the Massive Data Leak
ITPUB
ITPUB
Oct 5, 2025 · Backend Development

How to Clear a 10‑Million‑Message Queue in 5 Hours: A Five‑Step Rescue Plan

When a flash‑sale causes a 10 million‑message backlog and consumers only process 200 messages per second, this guide shows a five‑step, 5‑hour strategy—horizontal scaling, message downgrade, flow control, temporary dump, and parallel blasting—to restore throughput and prevent system collapse.

Distributed SystemsKafkaPerformance Optimization
0 likes · 6 min read
How to Clear a 10‑Million‑Message Queue in 5 Hours: A Five‑Step Rescue Plan
ITPUB
ITPUB
Oct 3, 2025 · Big Data

How Qunar Travel Cut 2000 CPU Cores by Optimizing Kafka Production

This case study details how Qunar Travel's engineering team analyzed Kafka production bottlenecks during peak traffic, added targeted monitoring, tuned thread and batch parameters, and validated the changes through gray‑scale tests, ultimately saving about 2000 CPU cores across three clusters while reducing request volume and improving network and disk utilization.

Big DataCPU SavingsKafka
0 likes · 14 min read
How Qunar Travel Cut 2000 CPU Cores by Optimizing Kafka Production
Cognitive Technology Team
Cognitive Technology Team
Oct 3, 2025 · Backend Development

Mastering Kafka Consumer Rebalance: Strategies to Boost Throughput and Stability

This article deeply explores Kafka consumer group rebalance mechanisms, identifies performance pitfalls of frequent rebalances, and provides a comprehensive set of configuration tweaks, assignment strategies, batch processing techniques, and monitoring practices to achieve a more stable and high‑throughput Kafka consumer system.

BackendConsumerKafka
0 likes · 16 min read
Mastering Kafka Consumer Rebalance: Strategies to Boost Throughput and Stability
IT Services Circle
IT Services Circle
Oct 1, 2025 · Backend Development

Designing a Fast, Reliable, Cost‑Effective Like System for High‑Traffic Apps

This article breaks down the essential requirements and architecture of a high‑performance like system, covering fast response, data consistency, scalability under traffic spikes, and cost‑efficient resource use, while detailing the technical stack, caching strategies, async persistence, and practical optimizations.

KafkaLike SystemScalability
0 likes · 17 min read
Designing a Fast, Reliable, Cost‑Effective Like System for High‑Traffic Apps
Architecture Digest
Architecture Digest
Sep 23, 2025 · Backend Development

How to Ensure Zero Message Loss in Kafka: Proven Strategies for High‑Reliability Systems

This article explains Kafka's storage architecture, identifies three major message‑loss scenarios across production, storage, and consumption, and provides practical end‑to‑end configurations, detection methods, and business‑level patterns to achieve near‑zero message loss in high‑concurrency distributed systems.

Data ConsistencyDistributed SystemsKafka
0 likes · 13 min read
How to Ensure Zero Message Loss in Kafka: Proven Strategies for High‑Reliability Systems
Su San Talks Tech
Su San Talks Tech
Sep 23, 2025 · Backend Development

How to Guarantee 100% Message Delivery with Kafka: Interview‑Ready Strategies

This article dissects Kafka’s storage architecture, identifies loss points in production, storage, and consumption phases, and presents interview‑ready strategies—including acks settings, flush tuning, consumer batch commits, detection via sequence numbers, and transactional messaging—to guarantee virtually 100 % message durability.

Consumer CommitKafkaReliability
0 likes · 20 min read
How to Guarantee 100% Message Delivery with Kafka: Interview‑Ready Strategies
Ray's Galactic Tech
Ray's Galactic Tech
Sep 21, 2025 · Cloud Native

How to Deploy a Scalable Kafka Cluster on Bare Metal and Kubernetes

This guide walks you through installing a high‑availability Kafka cluster, covering both bare‑metal/virtual‑machine setups and Kubernetes‑based deployments with Helm, including environment preparation, ZooKeeper configuration, broker settings, Helm chart customization, KRaft mode, and production‑grade tuning recommendations.

DeploymentKRaftKafka
0 likes · 7 min read
How to Deploy a Scalable Kafka Cluster on Bare Metal and Kubernetes
Su San Talks Tech
Su San Talks Tech
Sep 16, 2025 · Backend Development

Mastering Message Order in Distributed Queues: From Basics to Advanced Strategies

This article explores the fundamentals of message ordering in distributed message queues, explains why ordering is determined by broker arrival, compares global and partial ordering, and presents practical solutions—from single-partition designs to multi-partition hashing, handling data skew, and safe expansion—plus interview tips.

Distributed SystemsKafkaPartitioning
0 likes · 24 min read
Mastering Message Order in Distributed Queues: From Basics to Advanced Strategies
Mike Chen's Internet Architecture
Mike Chen's Internet Architecture
Sep 12, 2025 · Backend Development

How to Prevent Kafka Message Loss: Producer, Broker, and Consumer Strategies

This guide explains comprehensive Kafka message loss prevention techniques, covering producer configurations like acks=all, retries, idempotence, broker settings such as replication factor and min.insync.replicas, consumer handling with manual offset commits, and full‑stack compensation and retry mechanisms with code examples.

Broker SettingsConsumer OffsetKafka
0 likes · 3 min read
How to Prevent Kafka Message Loss: Producer, Broker, and Consumer Strategies
mikechen
mikechen
Sep 11, 2025 · Backend Development

How Does Kafka Guarantee Message Order? Key Mechanisms Explained

This article explains why message ordering is critical in scenarios such as financial transactions, e‑commerce order updates, audit logs, and IoT devices, then details Kafka’s core components—topics, partitions, offsets, producers, brokers, and consumer groups—and describes how single‑partition consumption and consistent key hashing ensure ordered processing while balancing throughput.

BackendKafkaMessage Ordering
0 likes · 5 min read
How Does Kafka Guarantee Message Order? Key Mechanisms Explained
Architect Chen
Architect Chen
Sep 10, 2025 · Big Data

How Kafka Achieves Million‑Message Throughput: Sequential Writes, Page Cache, Batching & Zero‑Copy

The article explains how Kafka attains high‑throughput performance by using sequential disk writes, leveraging the OS page cache, employing producer and consumer batching with configurable parameters, and utilizing zero‑copy sendfile to minimize CPU and memory overhead, enabling stable million‑message per second rates.

BatchingBig DataHigh Throughput
0 likes · 5 min read
How Kafka Achieves Million‑Message Throughput: Sequential Writes, Page Cache, Batching & Zero‑Copy
IT Architects Alliance
IT Architects Alliance
Sep 8, 2025 · Cloud Native

Choosing the Right Message Queue: Kafka vs RabbitMQ vs Pulsar – A Practical Guide

This article examines the core evaluation criteria for enterprise message queues and provides a detailed comparison of Kafka, RabbitMQ, and Pulsar—including performance, reliability, scalability, operational complexity, ecosystem maturity, and business fit—offering actionable guidance for selecting the optimal solution in cloud‑native environments.

KafkaPulsarRabbitMQ
0 likes · 9 min read
Choosing the Right Message Queue: Kafka vs RabbitMQ vs Pulsar – A Practical Guide
Ops Community
Ops Community
Sep 8, 2025 · Operations

Mastering Distributed Log Architecture: From Flume to ELK and Beyond

This comprehensive guide walks you through the challenges of large‑scale log collection, real‑time processing, storage optimization, and visualization, detailing practical configurations for Flume, Logstash, Elasticsearch, Kibana, Filebeat, Kafka, Kubernetes, and future AIOps integrations to build a reliable, cost‑effective distributed logging system.

ELKFlumeKafka
0 likes · 24 min read
Mastering Distributed Log Architecture: From Flume to ELK and Beyond
IT Services Circle
IT Services Circle
Sep 6, 2025 · Backend Development

10 Real‑World Scenarios Where Message Queues Transform Your System

This article explores ten practical use‑cases for message queues—covering system decoupling, asynchronous processing, traffic shaping, data synchronization, log collection, broadcast updates, ordered and delayed messages, retry mechanisms, and transactional messaging—illustrated with Java code examples and architectural diagrams.

Backend DevelopmentDistributed SystemsJava
0 likes · 17 min read
10 Real‑World Scenarios Where Message Queues Transform Your System
Mike Chen's Internet Architecture
Mike Chen's Internet Architecture
Sep 4, 2025 · Backend Development

How to Diagnose and Fix Kafka Message Backlog Issues

This article explains what Kafka message backlog is, why it harms performance and stability, and provides practical producer‑side throttling and consumer‑side scaling techniques, illustrated with diagrams, to help engineers eliminate backlog and keep their Kafka clusters healthy.

Kafkaconsumer scalingmessage backlog
0 likes · 4 min read
How to Diagnose and Fix Kafka Message Backlog Issues
Tech Freedom Circle
Tech Freedom Circle
Sep 4, 2025 · Backend Development

How to Solve ES Latency in MySQL‑Canal Sync and Indexing Scenarios?

The article dissects the interview question about ES latency in a MySQL‑Canal‑to‑Elasticsearch pipeline, explains the root causes across four system layers, and presents a comprehensive four‑layer optimization, end‑to‑end observability, routing‑based degradation, and a Java‑based LatencyProbe component to measure and control delay.

CanalElasticsearchKafka
0 likes · 17 min read
How to Solve ES Latency in MySQL‑Canal Sync and Indexing Scenarios?