BestHub
Sign in
Tagged articles

Kafka

1415 articles · Page 2 of 15
Cloud Architecture
Cloud Architecture
May 3, 2026 · Big Data

Cutting Log Storage Costs 70% for 100 Billion Daily Logs: A Full Guide to Hot‑Cold Separation Architecture

This article explains why massive log systems must adopt hot‑cold separation, walks through the problem analysis, SLO definition, component design, Kafka partition planning, Elasticsearch and ClickHouse tuning, Parquet archiving, a unified query gateway with async cold queries, governance practices, cost modeling, common pitfalls, and a roadmap for evolving the platform.

ElasticsearchKafkaLog Storage
0 likes · 40 min read
Cutting Log Storage Costs 70% for 100 Billion Daily Logs: A Full Guide to Hot‑Cold Separation Architecture
Java Tech Workshop
Java Tech Workshop
Apr 29, 2026 · Backend Development

How to Diagnose and Scale SpringBoot Message Backlog with Monitoring

The article explains why message backlog occurs in SpringBoot applications, outlines systematic troubleshooting steps, proposes comprehensive monitoring across producer, broker, and consumer layers, and presents scaling tactics such as instance expansion, concurrency tuning, batch consumption, and long‑term capacity planning.

BacklogKafkaRabbitMQ
0 likes · 16 min read
How to Diagnose and Scale SpringBoot Message Backlog with Monitoring
Java Tech Workshop
Java Tech Workshop
Apr 29, 2026 · Backend Development

Mastering SpringBoot Transactional Messaging for Distributed Consistency

This article explains how SpringBoot handles transactional messaging to achieve distributed data consistency, covering the concept of transaction messages, CAP theory, final consistency, and three practical implementations using a local message table, Kafka transactions, and RocketMQ, plus idempotency and compensation strategies.

CAP theoremDistributed ConsistencyKafka
0 likes · 17 min read
Mastering SpringBoot Transactional Messaging for Distributed Consistency
Java Tech Workshop
Java Tech Workshop
Apr 28, 2026 · Backend Development

Ensuring Message Order in SpringBoot: Partitioning and Sequential Consumption

This article examines why message ordering is critical in distributed systems, explains how partition mechanisms in Kafka, RocketMQ, and RabbitMQ enable ordered consumption, and provides detailed SpringBoot implementations, best‑practice guidelines, partition‑key design principles, concurrency settings, idempotency, and real‑world case studies to ensure reliable sequential processing.

Backend DevelopmentKafkaPartitioning
0 likes · 28 min read
Ensuring Message Order in SpringBoot: Partitioning and Sequential Consumption
Golang Shines
Golang Shines
Apr 28, 2026 · Backend Development

Essential Go Packages for Production Environments

This article compiles a curated list of production‑ready Go packages covering testing, logging, error handling, caching, databases, HTTP routing, HTTP clients, fault tolerance, Kafka, and various utility libraries, explaining their key features, concrete code examples, and why they are preferred in real‑world services.

GoHTTPKafka
0 likes · 15 min read
Essential Go Packages for Production Environments
LuTiao Programming
LuTiao Programming
Apr 28, 2026 · Backend Development

How I Built a High‑Performance Java Price‑Comparison Engine from Scratch

Starting from a simple sequential Java price‑aggregator, the article walks through successive architectural upgrades—concurrent calls with CompletableFuture, timeout and fallback handling, Spring Boot service exposure, caching, bulkhead isolation, microservice split, and Kafka‑driven event processing—showing how latency drops from 1500 ms to under 20 ms.

JavaKafkaMicroservices
0 likes · 9 min read
How I Built a High‑Performance Java Price‑Comparison Engine from Scratch
Java Tech Workshop
Java Tech Workshop
Apr 27, 2026 · Backend Development

How to Integrate Kafka with SpringBoot for High‑Performance Messaging

This article walks through Kafka’s core architecture, explains why it achieves massive throughput, and provides a step‑by‑step SpringBoot integration—including environment setup, Maven dependencies, configuration, producer and consumer code, advanced features like transactions and dead‑letter queues, plus performance monitoring and tuning tips.

ConsumerJavaKafka
0 likes · 11 min read
How to Integrate Kafka with SpringBoot for High‑Performance Messaging
DevOps Coach
DevOps Coach
Apr 26, 2026 · Backend Development

Forget Kafka: A Lightweight Go Queue Achieves 2 Million Messages per Second

The article analyzes how replacing Kafka with a simple in‑memory Go queue reduced architectural complexity, boosted throughput from 240‑330 K to 1.8‑2.0 M messages per second, and clarified debugging, while still acknowledging scenarios where Kafka remains the better choice.

Backend PerformanceGoIn‑Memory Ring Buffer
0 likes · 8 min read
Forget Kafka: A Lightweight Go Queue Achieves 2 Million Messages per Second
Cloud Architecture
Cloud Architecture
Apr 23, 2026 · Backend Development

Gracefully Stopping Java Threads: Kernel Mechanics to Cloud‑Native Guide

This article explains why Java threads cannot be force‑killed, how Thread.interrupt works, and provides a step‑by‑step framework—including cooperative cancellation, two‑phase termination, thread‑pool shutdown, Kafka consumer handling, Spring Boot lifecycle integration, and Kubernetes termination orchestration—to achieve safe, observable shutdown of production‑grade Java services.

Graceful ShutdownJavaKafka
0 likes · 31 min read
Gracefully Stopping Java Threads: Kernel Mechanics to Cloud‑Native Guide
LuTiao Programming
LuTiao Programming
Apr 22, 2026 · Backend Development

Kafka Deep Dive: Core Concepts Every Architect Must Master to Prevent Outages

The article explains why merely “knowing how to use” Kafka is insufficient, detailing how offset commits, consumer acknowledgments, producer acks, and rebalance behavior affect reliability, and provides concrete code examples, risk scenarios, and configuration recommendations to prevent message loss and duplicate processing in production systems.

Consumer offsetIdempotenceKafka
0 likes · 7 min read
Kafka Deep Dive: Core Concepts Every Architect Must Master to Prevent Outages
ITPUB
ITPUB
Apr 17, 2026 · Industry Insights

Why LinkedIn Dumped Kafka for Its Own ‘Northguard’ Streaming Engine

LinkedIn, the original home of Apache Kafka, abandoned the platform for a home‑grown system called Northguard, redesigning log storage, decentralizing metadata, and adding a virtualized Xinfra layer to handle trillions of daily events, while still acknowledging Kafka’s relevance for most companies.

InfrastructureKafkaLinkedIn
0 likes · 7 min read
Why LinkedIn Dumped Kafka for Its Own ‘Northguard’ Streaming Engine
Architect Chen
Architect Chen
Apr 16, 2026 · Big Data

Supercharge Kafka Consumer Performance: Parallelism, Batching, and Multithreading

This guide explains practical techniques to dramatically increase Kafka consumer throughput, including scaling consumer instances or partitions, tuning fetch and poll parameters, and implementing a multithreaded consumer model, while also covering hardware, JVM, and OS optimizations and monitoring recommendations.

Batch FetchKafkaPerformance Optimization
0 likes · 5 min read
Supercharge Kafka Consumer Performance: Parallelism, Batching, and Multithreading
Cloud Architecture
Cloud Architecture
Apr 13, 2026 · Operations

Ultimate Message Middleware Comparison: Choosing Kafka, RabbitMQ or RocketMQ

This article walks through why modern systems need a message broker, presents a four‑layer architectural view, compares Kafka, RabbitMQ and RocketMQ across design, performance, routing and transactional capabilities, and offers concrete scenario recommendations, engineering best‑practices, code samples and monitoring guidelines.

KafkaPerformanceRabbitMQ
0 likes · 33 min read
Ultimate Message Middleware Comparison: Choosing Kafka, RabbitMQ or RocketMQ
LuTiao Programming
LuTiao Programming
Apr 8, 2026 · Backend Development

From Chaos to Production: Building a Real Food-Delivery Backend with Spring Boot

The article chronicles the step‑by‑step evolution of a small team’s chaotic food‑delivery backend into a production‑ready system, detailing how they introduced layering, transactions, security, caching, async processing, messaging, observability, scalability, resilience, containerization, and testing using Spring Boot, Kafka, Redis, JWT, and Docker.

DockerKafkaMicroservices
0 likes · 10 min read
From Chaos to Production: Building a Real Food-Delivery Backend with Spring Boot
Ray's Galactic Tech
Ray's Galactic Tech
Apr 4, 2026 · Backend Development

How to Build a High‑Concurrency Story Creation Platform with AgentScope Java

This article presents a step‑by‑step engineering guide for constructing a production‑grade, high‑throughput story generation platform using AgentScope Java, Spring Boot, Kafka, Redis, PostgreSQL, and Kubernetes, covering architecture, task modeling, DAG orchestration, code organization, scalability, observability, and deployment best practices.

JavaKafkaMulti-agent
0 likes · 39 min read
How to Build a High‑Concurrency Story Creation Platform with AgentScope Java
Cloud Architecture
Cloud Architecture
Apr 3, 2026 · Operations

Zero‑Downtime High‑Throughput Kafka MirrorMaker 2 Migration with Dual‑Write and Rollback

This guide details a production‑grade Kafka MirrorMaker 2 migration for an e‑commerce order platform, covering zero‑downtime strategies, high‑throughput dual‑write, capacity planning, Kubernetes deployment, monitoring, offset continuity, gray‑scale cut‑over, rollback procedures, and extensive configuration examples.

DualWriteKafkaKubernetes
0 likes · 34 min read
Zero‑Downtime High‑Throughput Kafka MirrorMaker 2 Migration with Dual‑Write and Rollback
Alibaba Cloud Native
Alibaba Cloud Native
Mar 30, 2026 · Industry Insights

How Haier’s AIoT Platform Scaled to Billions of Messages with Kafka Serverless on Alibaba Cloud

The article details how Haier Smart Home’s AIoT platform tackled massive device messaging demands by migrating its self‑built Kafka clusters to Alibaba Cloud’s Kafka Serverless, outlining the technical challenges, step‑by‑step migration plan, custom performance tuning, risk‑co‑governance, and the resulting improvements in stability, throughput, and operational efficiency.

AIoTAlibaba CloudKafka
0 likes · 11 min read
How Haier’s AIoT Platform Scaled to Billions of Messages with Kafka Serverless on Alibaba Cloud
Cloud Architecture
Cloud Architecture
Mar 29, 2026 · Backend Development

Smart Parking Guidance Using Kafka, Flink, and Spring Boot

This article presents a senior architect’s end‑to‑end design of a smart parking guidance system, detailing how high‑frequency sensor streams are ingested via Spring Boot gateways, processed with Kafka and Flink for stateful de‑duplication, debouncing, and aggregation, and finally served through Redis, PostgreSQL and Spring Boot APIs for real‑time vehicle routing.

FlinkKafkaSmart Parking
0 likes · 35 min read
Smart Parking Guidance Using Kafka, Flink, and Spring Boot
Cloud Architecture
Cloud Architecture
Mar 29, 2026 · Backend Development

Kafka Ordered Writes and Reads: The Full Truth Behind Message Sequencing

This article demystifies Kafka's ordering guarantees, explains the difference between partition‑level, key‑level, and global ordering, and provides a complete, production‑grade guide—including architecture, configuration, code samples, and best‑practice patterns—to achieve reliable ordered processing while maintaining high throughput and scalability.

CDCConsumerExactly-Once
0 likes · 35 min read
Kafka Ordered Writes and Reads: The Full Truth Behind Message Sequencing
360 Zhihui Cloud Developer
360 Zhihui Cloud Developer
Mar 27, 2026 · Cloud Native

How AutoMQ Transforms Kafka into a Cloud‑Native, Elastic Messaging Service

This article examines the limitations of traditional Kafka in large‑scale deployments and presents AutoMQ’s cloud‑native redesign—detailing its stateless architecture, storage separation, automatic scaling, read/write isolation, performance benchmarks, and real‑world migration case studies that demonstrate reduced latency, higher throughput, and lower resource costs.

AutoMQCloud NativeElastic Scaling
0 likes · 13 min read
How AutoMQ Transforms Kafka into a Cloud‑Native, Elastic Messaging Service
Cloud Architecture
Cloud Architecture
Mar 22, 2026 · Backend Development

Message Queues Aren’t a Silver Bullet: Decoupling, Asynchrony & Peak‑Shaving

The article analyses why message queues are not a cure‑all, using real e‑commerce and finance incidents to dissect the three core benefits—decoupling, asynchronous processing, and peak‑shaving—while exposing production challenges such as loss, duplication, ordering, dead‑letter handling, and end‑to‑end consistency, and offering concrete Java code, benchmark tables and a practical MQ selection guide.

KafkaPeak ShavingRocketMQ
0 likes · 32 min read
Message Queues Aren’t a Silver Bullet: Decoupling, Asynchrony & Peak‑Shaving
Alibaba Cloud Native
Alibaba Cloud Native
Mar 20, 2026 · Cloud Native

How a Gaming Platform Scaled to Millions with RocketMQ & Kafka: A Cloud‑Native Success Story

Facing explosive growth, the game‑service platform 悠悠有品 rebuilt its architecture on Alibaba Cloud, using RocketMQ for core transaction messaging and Kafka for data synchronization, achieving elastic scaling, high availability, cost reduction, and reliable high‑concurrency processing across its trading and analytics pipelines.

Game PlatformKafkaRocketMQ
0 likes · 8 min read
How a Gaming Platform Scaled to Millions with RocketMQ & Kafka: A Cloud‑Native Success Story
dbaplus Community
dbaplus Community
Mar 5, 2026 · Backend Development

How to Ensure Message Order in Kafka: From Basics to Advanced Solutions

This article explains the concept of message ordering in distributed systems, details how Kafka stores messages in partitions, compares global and partial ordering, evaluates single‑partition, asynchronous, and multi‑partition solutions—including handling data skew and partition expansion—and provides a practical interview guide.

BackendKafkadistributed systems
0 likes · 22 min read
How to Ensure Message Order in Kafka: From Basics to Advanced Solutions
Architect Chen
Architect Chen
Mar 3, 2026 · Backend Development

Preventing Kafka Duplicate Consumption with Idempotent Design

This article explains practical strategies to avoid duplicate message consumption in Kafka, covering business idempotency with unique IDs, database or Redis deduplication tables, enabling producer idempotence, consumer-side checks, and Kafka's transaction-based exactly‑once semantics, along with their trade‑offs and suitable scenarios.

BackendExactly-OnceKafka
0 likes · 4 min read
Preventing Kafka Duplicate Consumption with Idempotent Design
Golang Shines
Golang Shines
Feb 25, 2026 · Big Data

Essential Kafka Core Concepts – A Comprehensive Cheat Sheet (PDF Included)

This article outlines Kafka's role as a leading MQ middleware, highlighting its key traits of decoupling, asynchronous processing, and throttling, compares it with traditional messaging systems, and provides a structured list of basic, intermediate, and advanced topics plus interview questions for in‑depth study.

Data IntegrationDistributed StreamingKafka
0 likes · 5 min read
Essential Kafka Core Concepts – A Comprehensive Cheat Sheet (PDF Included)
LuTiao Programming
LuTiao Programming
Feb 21, 2026 · Backend Development

Stop Building Chaotic Payment Systems: A Complete Design Methodology for Payment Domains

The article presents a production‑grade, modular design methodology for payment domain systems, detailing five core modules, essential principles such as idempotency and ACID, concrete implementation examples with Kafka, Redis, PostgreSQL, and guidance on risk, compliance, and high‑availability engineering.

Distributed ArchitectureKafkaPCI-DSS
0 likes · 8 min read
Stop Building Chaotic Payment Systems: A Complete Design Methodology for Payment Domains
ITPUB
ITPUB
Feb 11, 2026 · Backend Development

How to Guarantee Zero Message Loss in MQ Systems: A Full‑Lifecycle Design

This guide explains why guaranteeing 100% message reliability in MQ is a critical system‑design interview topic and presents a three‑layer architecture—production, storage, and consumption—detailing ACK settings, local message tables, broker replication, leader election safeguards, manual offset commits, and idempotent processing to prevent any message loss.

AcknowledgmentKafkaMQ
0 likes · 11 min read
How to Guarantee Zero Message Loss in MQ Systems: A Full‑Lifecycle Design
ITPUB
ITPUB
Feb 9, 2026 · Databases

ClickHouse vs Doris vs Redis: Real‑World Query Performance Test with Flink

Using a 600k‑record IP range dataset, we built identical tables in ClickHouse and Doris, and a Redis skip‑list store, then ran three Flink‑Kafka streaming jobs to compare query latency across the three databases under varying traffic rates, revealing Redis as fastest, ClickHouse second, Doris slowest.

ClickHouseDorisFlink
0 likes · 8 min read
ClickHouse vs Doris vs Redis: Real‑World Query Performance Test with Flink
Top Architect
Top Architect
Feb 4, 2026 · Backend Development

Build a Robust Asynchronous Processing SDK with Spring, Kafka and MySQL

This article introduces a generic asynchronous processing SDK for Java back‑ends, explaining its design principles, advantages, component architecture, database schema, configuration via Apollo, usage steps, and practical demonstrations, while providing complete code snippets and a GitHub repository for reference.

JavaKafkaSpring
0 likes · 12 min read
Build a Robust Asynchronous Processing SDK with Spring, Kafka and MySQL
Tencent Cloud Developer
Tencent Cloud Developer
Feb 4, 2026 · Backend Development

How We Cut Server Costs by 82%: Refactoring a High‑Concurrency QQ Game Service from C++ to Go with Kafka

This article details the redesign of a core QQ game achievement service that suffered from low resource utilization and heavy CAS contention, describing how moving from a synchronous C++ implementation to an asynchronous Go‑Kafka pipeline eliminated lock conflicts, reduced server count by 82%, and dramatically improved latency and stability.

Kafkahigh concurrencyrefactoring
0 likes · 11 min read
How We Cut Server Costs by 82%: Refactoring a High‑Concurrency QQ Game Service from C++ to Go with Kafka
Java Tech Enthusiast
Java Tech Enthusiast
Feb 3, 2026 · Backend Development

Spring Boot 4.0.2: Critical Kafka Fixes, Dependency Upgrades & What to Watch

Spring Boot 4.0.2 is a maintenance release that focuses on fixing over 20 bugs—including a critical Kafka transaction auto‑configuration issue—upgrading more than 40 core dependencies, improving documentation, and introducing minor breaking changes, making it essential for Java developers to upgrade promptly.

Kafkabug fixdependency-upgrade
0 likes · 8 min read
Spring Boot 4.0.2: Critical Kafka Fixes, Dependency Upgrades & What to Watch
LuTiao Programming
LuTiao Programming
Jan 30, 2026 · Backend Development

Spring Boot 4.0.2 Released: Over 20 Critical Bugs Fixed and Kafka Transaction Issue Resolved

Spring Boot 4.0.2, launched on January 22, brings more than 20 critical bug fixes, restores Kafka transaction auto‑configuration, improves startup performance by up to 15%, stabilizes native‑image support, refines test behavior, upgrades core dependencies, and introduces a few breaking changes, making it a must‑upgrade for production users.

KafkaNative ImagePerformance
0 likes · 8 min read
Spring Boot 4.0.2 Released: Over 20 Critical Bugs Fixed and Kafka Transaction Issue Resolved
Ray's Galactic Tech
Ray's Galactic Tech
Jan 30, 2026 · Cloud Native

Scale a Monolithic Article Interaction Service with Kubernetes Microservices

This article walks through converting a single‑service article interaction module—handling likes, favorites, and reads—into independent microservices deployed on Kubernetes, detailing architecture goals, service separation, Redis‑based high‑concurrency handling, Kafka async persistence, deployment configurations, auto‑scaling, and real‑world performance results.

KafkaRedis
0 likes · 8 min read
Scale a Monolithic Article Interaction Service with Kubernetes Microservices
Su San Talks Tech
Su San Talks Tech
Jan 30, 2026 · Backend Development

Why Kafka Rebalance Causes Backlog, Duplicates, and Data Loss—and How to Fix It

Kafka consumer group rebalances can trigger message backlogs, duplicate processing, and data loss; this article explains common rebalance triggers, their impact on consumption, and practical configuration and coding strategies—such as tuning timeout parameters, using manual offset commits, and sticky partition assignment—to minimize disruptions.

Consumer GroupData lossKafka
0 likes · 12 min read
Why Kafka Rebalance Causes Backlog, Duplicates, and Data Loss—and How to Fix It
LuTiao Programming
LuTiao Programming
Jan 27, 2026 · Big Data

Why LinkedIn Is Replacing Kafka with Its Own Next‑Gen Streaming System

LinkedIn, facing planetary‑scale data volumes, found Kafka’s architecture hitting fundamental limits and built Northguard—a decentralized, log‑striped streaming platform with Raft‑based metadata and an Xinfra migration layer—to gradually replace Kafka’s core responsibilities while maintaining compatibility.

KafkaLinkedInNorthguard
0 likes · 8 min read
Why LinkedIn Is Replacing Kafka with Its Own Next‑Gen Streaming System
DevOps Coach
DevOps Coach
Jan 27, 2026 · Backend Development

7 Essential Kafka Design Patterns Every Engineer Should Master

This guide presents seven practical Kafka design patterns—single‑key single‑write, log compaction, multi‑consumer‑group fan‑out, retry and dead‑letter topics, exactly‑once processing with Streams, schema evolution with Avro, and choreography vs orchestration—detailing when to use each, core principles, code examples, tips, common pitfalls, and final recommendations for building reliable, observable, and maintainable event‑driven systems.

Design PatternsEvent StreamingKafka
0 likes · 9 min read
7 Essential Kafka Design Patterns Every Engineer Should Master
Ray's Galactic Tech
Ray's Galactic Tech
Jan 23, 2026 · Backend Development

How to Build a Kafka‑Level High‑Performance Message Queue from Scratch

This article presents a step‑by‑step guide to designing and implementing a Kafka‑class distributed log‑based message queue kernel, covering architecture, sequential writes, sparse indexing, zero‑copy I/O, partitioning, replication, consumer‑group metadata, batch pipelines, crash recovery, and performance benchmarks.

KafkaLog StorageReplication
0 likes · 7 min read
How to Build a Kafka‑Level High‑Performance Message Queue from Scratch
Architect's Guide
Architect's Guide
Jan 22, 2026 · Big Data

Unlock Kafka’s Power: Core Concepts, High‑Performance Architecture & Real‑World Scaling Tips

This comprehensive guide explores Kafka’s core value as a message queue, explains producers, consumers, topics, partitions, and replication, dives into cluster architecture, zero‑copy I/O, resource planning for disks, memory, CPU and network, and provides practical configuration, consumer‑group management, and operational tooling tips for building high‑throughput, highly available Kafka deployments.

Kafkacluster scalingconsumer groups
0 likes · 31 min read
Unlock Kafka’s Power: Core Concepts, High‑Performance Architecture & Real‑World Scaling Tips
ITPUB
ITPUB
Jan 21, 2026 · Interview Experience

How to Design a Billion‑User Real‑Time Step Leaderboard for Interviews

This article breaks down the interview‑level system design of a WeChat‑style step leaderboard that must support over a billion users, handling massive write spikes, low‑latency friend ranking queries, storage scaling, and relationship complexity with a three‑part architecture using MQ, Redis, and MySQL.

KafkaLeaderboardRedis
0 likes · 8 min read
How to Design a Billion‑User Real‑Time Step Leaderboard for Interviews
Tech Freedom Circle
Tech Freedom Circle
Jan 15, 2026 · Backend Development

Kafka Rebalance Storm Crushed 120k QPS in JD Interview – How to Understand and Fix

In a JD senior Java architect interview, a Kafka consumer‑group rebalance storm caused QPS to drop from 120k to zero, triggering massive message loss and latency spikes, and the article walks through the rebalance fundamentals, failure causes, impact analysis, cooperative sticky assignor migration, and comprehensive monitoring and mitigation strategies.

Consumer GroupKafkaRebalance
0 likes · 28 min read
Kafka Rebalance Storm Crushed 120k QPS in JD Interview – How to Understand and Fix
Code Wrench
Code Wrench
Jan 14, 2026 · Backend Development

When to Choose NATS Over Kafka for Go Microservices: A Practical Guide

This article compares Kafka, RabbitMQ, and NATS for Go microservices, explains why Kafka is often over‑engineered for internal communication, and shows how NATS provides a lightweight, event‑driven alternative with concrete code examples and a clear selection matrix.

GoKafkaMicroservices
0 likes · 9 min read
When to Choose NATS Over Kafka for Go Microservices: A Practical Guide
JD Retail Technology
JD Retail Technology
Jan 13, 2026 · Backend Development

Deep Dive into Kafka, RocketMQ, and JMQ Storage Architectures

This article compares the storage models, data organization, indexing, read/write processes, and performance trade‑offs of three major message queues—Kafka, RocketMQ, and JMQ—providing detailed technical insights for architects and engineers making storage‑related design decisions.

Backend EngineeringJMQKafka
0 likes · 16 min read
Deep Dive into Kafka, RocketMQ, and JMQ Storage Architectures
Top Architect
Top Architect
Jan 12, 2026 · Backend Development

How to Build a Robust Asynchronous Processing SDK with Spring, Kafka, and XXL‑Job

This article explains the design and implementation of a generic asynchronous processing SDK for Java, covering its purpose, advantages, core principles, component choices, design patterns, configuration via Apollo, usage steps, safety considerations, and provides complete SQL and Spring configuration examples along with a GitHub repository link.

Design PatternsJavaKafka
0 likes · 11 min read
How to Build a Robust Asynchronous Processing SDK with Spring, Kafka, and XXL‑Job
iQIYI Technical Product Team
iQIYI Technical Product Team
Jan 8, 2026 · Big Data

How iQIYI Cut Stream Data Costs by 70%: From Private‑Cloud Kafka to AutoMQ

This article details iQIYI's evolution from a tightly coupled private‑cloud Kafka setup to a cloud‑native AutoMQ architecture, describing the challenges of scaling, the development of the Stream platform and Stream‑SDK, the migration to hybrid and public‑cloud Kafka, and the resulting cost and elasticity improvements.

AutoMQKafkacloud migration
0 likes · 12 min read
How iQIYI Cut Stream Data Costs by 70%: From Private‑Cloud Kafka to AutoMQ
Ray's Galactic Tech
Ray's Galactic Tech
Jan 6, 2026 · Backend Development

Scalable GPS Data Backend: SpringBoot, Kafka, MongoDB & Redis Design

This guide outlines a complete backend architecture for high‑volume GPS data, detailing the overall system flow, technology stack choices, Maven dependencies, data models, Kafka producer/consumer configurations, SpringBoot controllers, asynchronous processing, Redis caching, health checks, Docker deployment, and performance tuning recommendations to ensure stability and scalability.

DockerKafkaMongoDB
0 likes · 11 min read
Scalable GPS Data Backend: SpringBoot, Kafka, MongoDB & Redis Design
Tech Freedom Circle
Tech Freedom Circle
Jan 6, 2026 · Backend Development

Why Choose RocketMQ Over Kafka? The Real Reasons Behind the 90% Mistake

This article dissects a common interview question about Kafka's higher throughput versus RocketMQ's richer features, explains the underlying design philosophies, storage models, I/O paths, scaling limits, real‑world use cases such as transaction, delayed and ordered messages, and provides concrete optimization steps and code samples to help engineers make an informed messaging platform choice.

JavaKafkaPerformance
0 likes · 42 min read
Why Choose RocketMQ Over Kafka? The Real Reasons Behind the 90% Mistake
JakartaEE China Community
JakartaEE China Community
Jan 6, 2026 · Backend Development

Run a WildFly Application with JBang in Minutes

This tutorial shows how to use JBang scripts to launch a WildFly server, create a simple Jakarta REST endpoint, and extend the server with WildFly Glow add‑ons such as Kafka, providing step‑by‑step commands, code examples, and Docker integration for rapid prototyping.

JBangJakarta EEJava scripting
0 likes · 9 min read
Run a WildFly Application with JBang in Minutes
Architect Chen
Architect Chen
Jan 2, 2026 · Backend Development

Preventing Duplicate Consumption in Kafka: Design, Idempotence, and Configuration Strategies

This guide explains how to avoid duplicate message consumption in Kafka by designing unique identifiers, implementing consumer-side idempotence with deduplication tables, leveraging Kafka’s transactional features, and establishing system-level safeguards and monitoring to ensure reliable, exactly‑once processing.

Backend DevelopmentExactly-OnceIdempotence
0 likes · 4 min read
Preventing Duplicate Consumption in Kafka: Design, Idempotence, and Configuration Strategies
dbaplus Community
dbaplus Community
Dec 30, 2025 · Backend Development

How to Tackle Massive Message Queue Backlogs in High‑Traffic Scenarios

During peak traffic like Double‑11, a message queue can accumulate millions of messages, and simply adding consumer instances only offers temporary relief; this article explains the partition model limits, how to calculate proper partition numbers, fast remediation tactics, and deep consumer‑side optimizations for robust, scalable processing.

BacklogKafkaconsumer optimization
0 likes · 20 min read
How to Tackle Massive Message Queue Backlogs in High‑Traffic Scenarios
Mike Chen's Internet Architecture
Mike Chen's Internet Architecture
Dec 25, 2025 · Backend Development

How to Resolve Kafka Backlog Under High Load: Practical Tips

This article explains why Kafka experiences message backlog in high‑load environments, identifies producer‑consumer speed mismatches, I/O and resource bottlenecks, and offers concrete strategies such as scaling consumers, tuning hardware, and adjusting Kafka configurations to eliminate the backlog.

BacklogKafkadistributed systems
0 likes · 4 min read
How to Resolve Kafka Backlog Under High Load: Practical Tips
Mike Chen's Internet Architecture
Mike Chen's Internet Architecture
Dec 22, 2025 · Operations

How to Diagnose and Resolve Kafka Consumer Lag Quickly

When Kafka consumers fall behind, this guide walks you through confirming the backlog, pinpointing bottlenecks in production, consumption, or brokers, and applying concrete steps—such as checking offsets, comparing TPS, inspecting consumer logic, and adjusting partitions—to efficiently eliminate lag.

Consumer LagKafka
0 likes · 4 min read
How to Diagnose and Resolve Kafka Consumer Lag Quickly
Architect Chen
Architect Chen
Dec 21, 2025 · Backend Development

How to Resolve Kafka Backlog: Boost Consumer Throughput and Optimize Partitions

This guide explains why Kafka backlog occurs when production outpaces consumption and provides practical steps—such as increasing consumer instances, optimizing processing, expanding partitions, applying flow‑control, and managing message capacity—to eliminate the backlog and keep the cluster healthy.

BacklogFlow ControlKafka
0 likes · 4 min read
How to Resolve Kafka Backlog: Boost Consumer Throughput and Optimize Partitions
Mike Chen's Internet Architecture
Mike Chen's Internet Architecture
Dec 9, 2025 · Backend Development

Boost Kafka to Over 1 Million Messages per Second: Metrics and Tuning Tips

This article explains what high concurrency means for Kafka, outlines key performance metrics such as QPS, TPS, throughput and latency, and provides concrete configuration and architectural techniques—including broker optimization, horizontal scaling, network batching, and zero‑copy—to achieve write rates exceeding one million records per second.

BackendKafkadistributed systems
0 likes · 4 min read
Boost Kafka to Over 1 Million Messages per Second: Metrics and Tuning Tips
IT Services Circle
IT Services Circle
Dec 9, 2025 · Backend Development

Mastering Kafka Rebalance: Prevent Backlog, Duplicates, and Data Loss

When Kafka consumer groups rebalance, partitions are reassigned, often causing message backlog, duplicate processing, or loss; understanding the triggers, impact, and optimization techniques—like tuning timeouts, managing offset commits, and using sticky assignors—can keep your streaming pipelines reliable.

Consumer GroupKafkaRebalance
0 likes · 13 min read
Mastering Kafka Rebalance: Prevent Backlog, Duplicates, and Data Loss
dbaplus Community
dbaplus Community
Dec 8, 2025 · Databases

Which Database Wins IP Range Lookups? ClickHouse vs Doris vs Redis Benchmarks

This article presents a systematic benchmark comparing ClickHouse, Doris, and Redis for IP‑range dimension lookups using Flink‑Kafka pipelines, detailing test design, result table schema, query interfaces, and performance results across varying data rates, concluding that Redis offers the fastest and most stable query latency.

ClickHouseDatabase BenchmarkDoris
0 likes · 7 min read
Which Database Wins IP Range Lookups? ClickHouse vs Doris vs Redis Benchmarks
Mike Chen's Internet Architecture
Mike Chen's Internet Architecture
Dec 3, 2025 · Big Data

Kafka High‑Throughput Tricks: Sequential Writes, Zero‑Copy, Partitioning

The article explains how Kafka achieves high throughput by writing messages sequentially to disk, leveraging OS page cache and zero‑copy system calls, using partitioned topics for parallelism, batching and compressing records on both producer and broker sides, and employing asynchronous replication with configurable persistence strategies.

CompressionKafkaPartitioning
0 likes · 4 min read
Kafka High‑Throughput Tricks: Sequential Writes, Zero‑Copy, Partitioning
Code Wrench
Code Wrench
Nov 22, 2025 · Backend Development

Build a Production-Ready Rule Engine with Gray Release Using Go, Kafka, and Redis

Learn how to design and implement a ready-to-use rule engine combined with a gray release system using Golang, Kafka, Redis, and CEL, complete with Docker‑compose deployment, edge execution, token‑bucket throttling, and webhook actions, plus full source code for a production‑grade marketing strategy platform.

CELGoGray Release
0 likes · 9 min read
Build a Production-Ready Rule Engine with Gray Release Using Go, Kafka, and Redis
dbaplus Community
dbaplus Community
Nov 18, 2025 · Backend Development

How to Guarantee 100% No Message Loss in Distributed MQ Systems

Ensuring that messages never disappear in a distributed MQ system requires a three‑pronged strategy covering production, storage, and consumption, with proper ACK configurations, local message tables, replication settings, and manual offset commits to achieve reliable, at‑least‑once processing without data loss.

BackendKafkaMQ
0 likes · 11 min read
How to Guarantee 100% No Message Loss in Distributed MQ Systems
MaGe Linux Operations
MaGe Linux Operations
Nov 18, 2025 · Big Data

Zero‑Data‑Loss Kafka Cluster Scaling: Complete Step‑by‑Step Guide

This comprehensive guide explains how to safely expand a Kafka cluster without data loss by covering applicable scenarios, pre‑conditions, anti‑pattern warnings, environment matrices, a detailed checklist, step‑by‑step Linux commands for broker preparation, partition‑rebalancing plan generation, throttled execution, real‑time monitoring, verification, rollback procedures, backup strategies, performance testing, common troubleshooting, FAQs and best‑practice scripts, all illustrated with code snippets and practical examples.

KafkaLinuxPartition Rebalancing
0 likes · 47 min read
Zero‑Data‑Loss Kafka Cluster Scaling: Complete Step‑by‑Step Guide
Ray's Galactic Tech
Ray's Galactic Tech
Nov 17, 2025 · Backend Development

Kafka vs RabbitMQ vs RocketMQ: Which Message Broker Fits Your Use Case?

This article provides an in‑depth, multi‑dimensional comparison of Apache Kafka, RabbitMQ, and Apache RocketMQ—covering design philosophy, performance, reliability, features, ecosystem, operations, and typical scenarios—to help you choose the right message middleware for your architecture.

KafkaRabbitMQRocketMQ
0 likes · 6 min read
Kafka vs RabbitMQ vs RocketMQ: Which Message Broker Fits Your Use Case?
Tech Freedom Circle
Tech Freedom Circle
Nov 15, 2025 · Databases

How to Prevent Order Loss in a 100k TPS Flash Sale When the Master DB Crashes – 5 Practical Solutions

The article dissects a high‑traffic flash‑sale interview question—how to guarantee zero order loss at 100,000 TPS when the master MySQL instance fails—by explaining the underlying performance‑consistency conflict, the three skills interviewers assess, and presenting five concrete, code‑driven solutions ranging from MySQL parameter tuning to semi‑sync replication, local message tables, group replication, and Redis‑Kafka traffic shaping.

Group ReplicationKafkaMySQL
0 likes · 28 min read
How to Prevent Order Loss in a 100k TPS Flash Sale When the Master DB Crashes – 5 Practical Solutions
mikechen
mikechen
Nov 13, 2025 · Backend Development

How to Diagnose and Resolve Kafka Message Backlog Issues

This article explains what Kafka message backlog is, outlines the main reasons it occurs—such as producer speed outpacing consumers, slow consumer processing, and downstream bottlenecks—and provides practical steps for producer throttling, consumer scaling and logic improvements, and Kafka cluster enhancements to eliminate the backlog.

Backend DevelopmentKafkamessage backlog
0 likes · 6 min read
How to Diagnose and Resolve Kafka Message Backlog Issues
Ray's Galactic Tech
Ray's Galactic Tech
Nov 9, 2025 · Backend Development

Hybrid Push‑Pull Timeline Architecture: Scaling Social Feeds for Billions

To serve billions of users with real‑time timelines, modern social platforms combine push‑based delivery for regular users and pull‑based retrieval for high‑profile accounts, employing hot‑cold separation, Kafka fan‑out, Redis caching, and scalable storage strategies to balance write and read loads.

KafkaPush‑PullRedis
0 likes · 9 min read
Hybrid Push‑Pull Timeline Architecture: Scaling Social Feeds for Billions
Ops Community
Ops Community
Nov 6, 2025 · Big Data

Zero Data Loss Kafka Cluster Scaling: From 3 to 10 Nodes – A Complete Guide

This comprehensive guide walks you through expanding or shrinking a production‑grade Kafka cluster—covering prerequisites, anti‑pattern warnings, environment matrices, step‑by‑step expansion and contraction procedures, partition rebalancing principles, monitoring, best practices, and troubleshooting—to ensure zero data loss during scaling.

KafkaPartition RebalancingZero Data Loss
0 likes · 27 min read
Zero Data Loss Kafka Cluster Scaling: From 3 to 10 Nodes – A Complete Guide
DataFunTalk
DataFunTalk
Nov 6, 2025 · Cloud Native

How Tencent Music Cut Kafka Costs by 50% with Cloud‑Native AutoMQ

Tencent Music migrated its massive Kafka streaming infrastructure to the cloud‑native AutoMQ platform, slashing operational costs by over half, achieving second‑level partition migration, and dramatically improving scaling efficiency while maintaining high‑throughput, low‑latency data processing for its music services.

AutoMQData StreamingKafka
0 likes · 16 min read
How Tencent Music Cut Kafka Costs by 50% with Cloud‑Native AutoMQ
Top Architect
Top Architect
Oct 31, 2025 · Backend Development

Mastering Message Queues: A Deep Dive into RabbitMQ, RocketMQ, and Kafka

This comprehensive guide explains the core components, exchange types, TTL, confirm mechanisms, consumer ACK/NACK, dead‑letter queues, and high‑availability features of RabbitMQ, RocketMQ, and Kafka, while also covering load balancing, ordering, transaction handling, and best practices for reliable message delivery.

Backend DevelopmentKafkaRabbitMQ
0 likes · 32 min read
Mastering Message Queues: A Deep Dive into RabbitMQ, RocketMQ, and Kafka
Big Data Technology & Architecture
Big Data Technology & Architecture
Oct 30, 2025 · Backend Development

What’s New in Apache Kafka 4.1? Core Features and Architecture Changes Explained

Apache Kafka 4.1.0 introduces native queue semantics, a new Streams rebalancing protocol, multi‑version Connect plugins, a revamped consumer‑group protocol, enhanced transaction safety, and numerous client, monitoring, and security improvements, offering a comprehensive upgrade over the 4.0 release.

KafkaStreamingdistributed-systems
0 likes · 6 min read
What’s New in Apache Kafka 4.1? Core Features and Architecture Changes Explained
Su San Talks Tech
Su San Talks Tech
Oct 28, 2025 · Backend Development

How to Prevent MQ Message Loss: 5 Proven Strategies for Reliable Messaging

Discover the three stages where MQ messages can be lost and explore five practical solutions—including producer confirmations, message persistence, consumer acknowledgments, transactional messaging, and retry with dead‑letter queues—complete with code examples and guidance on selecting the right approach for different scenarios.

KafkaRabbitMQTransactional Messaging
0 likes · 14 min read
How to Prevent MQ Message Loss: 5 Proven Strategies for Reliable Messaging
Shepherd Advanced Notes
Shepherd Advanced Notes
Oct 24, 2025 · Backend Development

Why Choose Spring Boot + DelayQueue for a Custom Distributed Delayed-Task Queue?

The article systematically analyzes common distributed delayed‑task implementations—Redis ZSet scanning, message‑queue delay features, and Redis key‑expiration listeners—highlighting their pros, cons, and suitable scenarios, then proposes a Spring Boot + DelayQueue component to achieve precise timing, dynamic delays, and robust coordination.

DelayQueueKafkaRedis
0 likes · 11 min read
Why Choose Spring Boot + DelayQueue for a Custom Distributed Delayed-Task Queue?
Huolala Tech
Huolala Tech
Oct 22, 2025 · Backend Development

Scaling Real‑Time Reconciliation with Dynamic Kafka Consumer Clusters

To ensure fund safety and robust operations, the team built a real‑time reconciliation platform that leverages Kafka, and after encountering scaling bottlenecks with a static consumer model, they implemented a dynamic, partition‑level, weighted load‑balancing consumer cluster that supports automatic scaling and high‑throughput processing.

Dynamic ScalingKafkabackend architecture
0 likes · 15 min read
Scaling Real‑Time Reconciliation with Dynamic Kafka Consumer Clusters
Architect Chen
Architect Chen
Oct 22, 2025 · Big Data

How to Eliminate Kafka Message Backlog with Practical Optimizations

This guide presents concrete techniques for improving Kafka consumer and producer performance, scaling clusters, tuning broker settings, and designing asynchronous buffering layers to prevent message accumulation and boost overall throughput.

KafkaPerformance OptimizationStreaming
0 likes · 5 min read
How to Eliminate Kafka Message Backlog with Practical Optimizations
Raymond Ops
Raymond Ops
Oct 21, 2025 · Big Data

Deep Dive into Kafka Architecture: Topics, Partitions, and Reliable Data Pipelines

This article explains Kafka’s core concepts—including topics, partitions, log segmentation, indexing, and acknowledgment mechanisms—then provides a step‑by‑step guide to deploy a Zookeeper‑Kafka cluster integrated with Filebeat, Logstash, and the ELK stack for reliable log collection and analysis.

ELKFilebeatKafka
0 likes · 11 min read
Deep Dive into Kafka Architecture: Topics, Partitions, and Reliable Data Pipelines
Code Wrench
Code Wrench
Oct 18, 2025 · Backend Development

Master High‑Performance Queues in Go: Kafka, RabbitMQ & Redis Compared

This article explains how to build a high‑throughput, low‑latency, and scalable queue system in Go by leveraging Kafka, RabbitMQ, and Redis, covering core concepts, practical code examples, performance optimizations, and guidance on choosing the right solution for different workloads.

GoKafkaRabbitMQ
0 likes · 11 min read
Master High‑Performance Queues in Go: Kafka, RabbitMQ & Redis Compared
Programmer XiaoFu
Programmer XiaoFu
Oct 16, 2025 · Operations

Stop Guessing: Kafka Message Backlog, Duplicates, and Loss Are Usually Caused by Rebalance

Kafka consumer issues such as message backlog, duplicate processing, and data loss often stem from consumer group rebalances triggered by changes in consumer count, partition count, subscription topics, or heartbeat and poll timeouts, and can be mitigated by tuning timeout settings, managing offset commits, and using sticky partition assignment.

Consumer GroupKafkaMessage Duplication
0 likes · 13 min read
Stop Guessing: Kafka Message Backlog, Duplicates, and Loss Are Usually Caused by Rebalance
Ubiquitous Tech
Ubiquitous Tech
Oct 15, 2025 · Backend Development

Java Interview Simulation: Solving the Architecture Challenges of a Hotel Pricing System

This article walks through a detailed Java interview simulation where the candidate designs a high‑throughput hotel pricing service, covering business decomposition, Kafka event‑driven pipelines, cache‑aside consistency, HikariCP tuning, fault‑tolerant microservices, multi‑region disaster recovery, and AI recommendation gray‑release strategies.

ArchitectureJavaKafka
0 likes · 43 min read
Java Interview Simulation: Solving the Architecture Challenges of a Hotel Pricing System
StarRocks
StarRocks
Oct 14, 2025 · Big Data

How Ctrip Scaled UBT Analytics by Migrating from ClickHouse to StarRocks

Ctrip's User Behavior Tracking (UBT) system, handling 30 TB of daily data, moved from ClickHouse to StarRocks' compute‑storage separated architecture, cutting average query latency from 1.4 seconds to 203 ms, halving storage, reducing nodes from 50 to 40, and boosting write throughput to 3 million rows per second.

ClickHouseFlinkKafka
0 likes · 15 min read
How Ctrip Scaled UBT Analytics by Migrating from ClickHouse to StarRocks
Smart Sea Tide
Smart Sea Tide
Oct 13, 2025 · Big Data

How Zhihu Evolved Its Real-Time Data Warehouse Architecture

The article details Zhihu's step‑by‑step evolution from a Spark‑Streaming‑based real‑time data warehouse (v1.0) to a Flink‑Streaming SQL‑driven platform (v2.0), explaining design choices, Lambda architecture implementation, scalability challenges, and future automation plans.

DruidFlinkHBase
0 likes · 22 min read
How Zhihu Evolved Its Real-Time Data Warehouse Architecture