Skip to content
Tagged articles

Kafka

500 articles · Page 2 of 5
Alibaba Cloud Developer
Alibaba Cloud Developer
Jun 18, 2026 · Big Data

How AI-Driven Real-Time Data Lakes Are Ditching ETL: A Kafka‑to‑Iceberg Architecture Simplification

In the AI era, enterprises need a data foundation that supports both low‑latency streaming and long‑term analytics, and the combination of Kafka, Iceberg and object storage is emerging as a preferred solution; by moving ingestion capabilities closer to the message layer and eliminating external ETL jobs, a "zero‑ETL" approach reduces architectural complexity, improves consistency, and streamlines schema evolution and small‑file management.

CDCIcebergKafka
0 likes · 27 min read
How AI-Driven Real-Time Data Lakes Are Ditching ETL: A Kafka‑to‑Iceberg Architecture Simplification
Random Bulletin
Random Bulletin
Jun 17, 2026 · Backend Development

Rebalance Optimization for Million‑QPS Kafka Clusters: From Default to Custom Assignors

At a 5,000‑instance, 20,000‑partition Kafka consumer cluster, the default eager Rebalance caused a 90‑second stop‑the‑world pause, illustrating how a single GC‑induced heartbeat miss can flood brokers with tens of millions of messages; the article dissects eager vs cooperative protocols, built‑in assignor limits, and four custom assignor patterns to scale Rebalance from 100 K to tens of millions of QPS.

AssignorCooperative ProtocolCustom Scheduler
0 likes · 28 min read
Rebalance Optimization for Million‑QPS Kafka Clusters: From Default to Custom Assignors
Random Bulletin
Random Bulletin
Jun 16, 2026 · Backend Development

Consumer Rate Limiting: From Zero to Full Control in Million‑QPS Architectures

The article explains why consumer‑side rate limiting is essential in million‑QPS systems, detailing how unchecked consumers can overwhelm downstream services, and presents practical strategies—including pause/resume, token‑bucket algorithms, adaptive thresholds, and global coordination—to safely throttle consumption without dropping messages.

Distributed SystemsKafkaMessage Queue
0 likes · 16 min read
Consumer Rate Limiting: From Zero to Full Control in Million‑QPS Architectures
Random Bulletin
Random Bulletin
Jun 15, 2026 · Backend Development

From Auto to Manual: Mastering Consumer Offsets for Ten‑Million QPS Systems

The article examines a real payment loss incident caused by Kafka's default automatic offset commit, explains why automatic commits become a hidden trap at high traffic, and provides a step‑by‑step guide to switching to manual commits with async‑first, sync‑fallback, idempotency, dead‑letter handling, batch strategies, and concurrency controls for reliable ten‑million QPS consumption.

IdempotencyKafkaMessage Queue
0 likes · 15 min read
From Auto to Manual: Mastering Consumer Offsets for Ten‑Million QPS Systems
Cloud Architecture
Cloud Architecture
Jun 15, 2026 · Backend Development

Trillion‑Message Engine Showdown: RabbitMQ vs Kafka Architecture, Performance and Cloud‑Native Pitfalls

An experienced architect compares RabbitMQ and Kafka across core protocols, storage, replication, consumption semantics, and real‑world production designs, offering Java 17/Spring Boot code, cloud‑native deployment tips, observability, and a decision framework that matches messaging patterns to business requirements.

Cloud NativeJavaKafka
0 likes · 45 min read
Trillion‑Message Engine Showdown: RabbitMQ vs Kafka Architecture, Performance and Cloud‑Native Pitfalls
Random Bulletin
Random Bulletin
Jun 14, 2026 · Backend Development

Scaling to Millions of QPS: How Batch Consumption Beats Single-Message Processing

The article explains why single‑message consumption stalls under high QPS due to fixed per‑message overhead, and how merging pull, processing, and commit into batch operations dramatically boosts throughput while introducing trade‑offs in latency, memory, and failure handling, with practical guidelines for batch size selection and dynamic tuning.

Batch ConsumptionKafkaMessage Queue
0 likes · 16 min read
Scaling to Millions of QPS: How Batch Consumption Beats Single-Message Processing
Linyb Geek Road
Linyb Geek Road
Jun 14, 2026 · Big Data

How to Solve Data Ordering Issues in Apache Kafka

This article explains how Kafka maintains order within partitions using keys and offsets, why ordering across partitions can break, and how to preserve strict sequencing through producer configuration, idempotent producers, and exactly‑once transactional processing.

Kafkadata orderingexactly-once
0 likes · 9 min read
How to Solve Data Ordering Issues in Apache Kafka
Random Bulletin
Random Bulletin
Jun 12, 2026 · Backend Development

From Random to Intelligent: Partition Selection Strategies for 10 Million QPS Systems

The article walks through four generations of partition‑selection strategies—from naive random and round‑robin to key‑hash, sticky, and finally intelligent load‑aware routing—explaining how hotspot keys, batch inefficiency, consumer skew, and fault amplification threaten stability at 10 M QPS and offering concrete engineering actions to design, test, monitor, and switch strategies in large‑scale message‑queue deployments.

KafkaPartitioninghigh‑throughput
0 likes · 20 min read
From Random to Intelligent: Partition Selection Strategies for 10 Million QPS Systems
Random Bulletin
Random Bulletin
Jun 10, 2026 · Backend Development

Message Compression at Scale: From No Compression to Selective Strategies

A midnight bandwidth alarm triggers a deep dive into message compression, revealing how algorithm choice, compression placement, batch handling, and selective policies evolve from simple no‑compression setups to multi‑layered strategies that balance CPU, bandwidth, disk costs, and latency in million‑QPS systems.

KafkaZstdbatch compression
0 likes · 20 min read
Message Compression at Scale: From No Compression to Selective Strategies
Random Bulletin
Random Bulletin
Jun 9, 2026 · Backend Development

From Synchronous to Asynchronous Sending: Achieving 10 Million QPS with Kafka Producer

The article analyses how synchronous Kafka producer calls block business threads on ACK, limiting throughput, and shows that moving to asynchronous sending with the Accumulator decouples submission from confirmation, unlocking batch, compression, and back‑pressure mechanisms that enable stable operation at tens of millions of QPS.

Kafkaasynchronousbackpressure
0 likes · 21 min read
From Synchronous to Asynchronous Sending: Achieving 10 Million QPS with Kafka Producer
Architect Chen
Architect Chen
Jun 9, 2026 · Big Data

How Kafka Prevents Duplicate Consumption: Three Main Solutions

The article explains why Kafka does not guarantee exactly‑once delivery and presents three practical approaches—business‑level idempotence, manual offset management, and Kafka’s transaction/EOS features—to reliably avoid duplicate message processing.

ConsumerKafkaTransaction
0 likes · 4 min read
How Kafka Prevents Duplicate Consumption: Three Main Solutions
Java Tech Enthusiast
Java Tech Enthusiast
Jun 6, 2026 · Backend Development

Why Kafka Marks a Live Consumer as Dead and Forces Rebalance

Even when a consumer process runs and logs normally, Kafka may deem it dead and trigger a rebalance because the poll interval exceeds max.poll.interval.ms, a situation known as ‘false dead’; this article explains the root cause and practical ways to prevent it.

ConsumerKafkaMultithreading
0 likes · 8 min read
Why Kafka Marks a Live Consumer as Dead and Forces Rebalance
Architectural Methodology
Architectural Methodology
Jun 6, 2026 · Interview Experience

How to Plan a Phased Migration and Ensure Data Consistency for Legacy System Refactoring

The article outlines a step‑by‑step architecture redesign for a 20‑year‑old monolithic production system, detailing problem analysis, a phased microservice migration using the "killer" pattern, data‑sync strategies, distributed transaction handling, observability, rollback mechanisms, and practical interview‑style insights.

CanalCloud NativeDDD
0 likes · 15 min read
How to Plan a Phased Migration and Ensure Data Consistency for Legacy System Refactoring
Cloud Architecture
Cloud Architecture
Jun 4, 2026 · Backend Development

Kafka Backlog Mastery: Root Causes, Emergency Fixes, and Production‑Grade Governance

This comprehensive guide explains why Kafka message backlog occurs, how to diagnose its root causes, and provides a step‑by‑step 5‑minute emergency response and production‑grade consumer architecture, including back‑pressure control, idempotent processing, capacity planning, observability, and cloud‑native deployment strategies.

BacklogConsumerKafka
0 likes · 48 min read
Kafka Backlog Mastery: Root Causes, Emergency Fixes, and Production‑Grade Governance
Random Bulletin
Random Bulletin
Jun 3, 2026 · Operations

Designing Partitions for Millions of QPS: From Default Settings to Precise Capacity Planning

The article explains why partition count is not a simple integer but the solution of a multi‑constraint capacity‑planning equation, walks through three design stages—from naïve defaults, through CPU‑core‑based heuristics, to a precise engineering model that balances throughput, ordering, scaling, fault domains and storage costs for million‑plus QPS workloads.

Fault ToleranceKafkaPartition Design
0 likes · 32 min read
Designing Partitions for Millions of QPS: From Default Settings to Precise Capacity Planning
Coder Trainee
Coder Trainee
Jun 3, 2026 · Cloud Native

Unified Messaging with Spring Cloud Stream: One Codebase for Multiple MQs

This article demonstrates how Spring Cloud Stream provides a unified programming model that decouples business logic from the underlying message broker, allowing a single codebase to work with RocketMQ, Kafka, or RabbitMQ by only changing configuration, and walks through project structure, implementation, conditional routing, MQ switching, testing, and common pitfalls.

JavaKafkaMessage-Driven
0 likes · 19 min read
Unified Messaging with Spring Cloud Stream: One Codebase for Multiple MQs
TechVision Expert Circle
TechVision Expert Circle
Jun 2, 2026 · Artificial Intelligence

How to Build a System with True Self‑Decision Capability

This article details a CTO‑driven initiative to give core business systems genuine self‑decision ability by defining the concept, presenting a four‑layer architecture, and sharing concrete design choices, safety mechanisms, governance practices, and real‑world lessons learned from an e‑commerce fulfillment use case.

GovernanceKafkaLLM agents
0 likes · 14 min read
How to Build a System with True Self‑Decision Capability
Random Bulletin
Random Bulletin
Jun 2, 2026 · Backend Development

Queue Selection at Ten‑Million QPS: From General‑Purpose to Specialized Queues

In systems handling ten‑million QPS, generic message queues like Kafka hit scalability and latency limits, so the article breaks down a three‑question framework—loss tolerance, latency sensitivity, and ordering needs—to match five specialized queue types and guide a transition from a single‑queue approach to a queue matrix.

Distributed SystemsKafkaMessage Queue
0 likes · 18 min read
Queue Selection at Ten‑Million QPS: From General‑Purpose to Specialized Queues
StarRocks
StarRocks
May 28, 2026 · Industry Insights

How Fresha Built a Modern Real‑Time Analytics Stack with AutoMQ and StarRocks

Fresha replaced its Postgres‑Snowflake‑MSK pipeline with an AutoMQ‑based Diskless Kafka message layer and StarRocks for real‑time analytics, cutting storage costs 17‑20×, dropping query latency from seconds to sub‑second, and migrating ~1,000 topics in a week with zero downtime.

AutoMQCloud MigrationData Pipeline
0 likes · 24 min read
How Fresha Built a Modern Real‑Time Analytics Stack with AutoMQ and StarRocks
Cloud Architecture
Cloud Architecture
May 22, 2026 · Operations

Kafka vs Pulsar: Choosing the Right Messaging System for Architecture and Production

Kafka and Pulsar are both mature distributed messaging platforms, but they differ fundamentally in architecture, performance, scalability, and operational complexity; this article analyzes their core designs, benchmarks, engineering trade‑offs, and real‑world scenarios to guide architects in making a reliable technology selection.

KafkaPerformancePulsar
0 likes · 36 min read
Kafka vs Pulsar: Choosing the Right Messaging System for Architecture and Production
LuTiao Programming
LuTiao Programming
May 21, 2026 · Backend Development

Stop Fighting Microservice Calls—Why Experts Prefer Event‑Driven Architecture for Decoupling Distributed Systems

The article explains how traditional synchronous microservice calls create tight coupling, cascading failures, scaling bottlenecks, and high latency, and demonstrates that adopting an event‑driven architecture with producers, consumers, and a message broker such as Kafka can fully decouple services, improve scalability, and enable patterns like event sourcing and CQRS.

CQRSDistributed SystemsKafka
0 likes · 14 min read
Stop Fighting Microservice Calls—Why Experts Prefer Event‑Driven Architecture for Decoupling Distributed Systems
Su San Talks Tech
Su San Talks Tech
May 19, 2026 · Interview Experience

Designing a Hundred‑Billion‑Scale Message Queue: A ByteDance Interview Walkthrough

This article walks through the interview question of designing a message queue that handles billions of messages daily and peaks at millions of QPS, covering traffic calculations, core roles, storage and throughput techniques, scalability, high availability, observability, framework comparisons, a real‑world case study, and key follow‑up interview topics.

KafkaMessage QueuePulsar
0 likes · 12 min read
Designing a Hundred‑Billion‑Scale Message Queue: A ByteDance Interview Walkthrough
Cloud Architecture
Cloud Architecture
May 9, 2026 · Backend Development

High‑Concurrency Flash‑Sale Blueprint: Redis Atomic Stock and Kafka Throttling

The article presents a production‑grade, scalable flash‑sale architecture that combines Redis atomic inventory deduction, Kafka asynchronous peak‑shaving, and careful database finalization, detailing each layer’s goals, pre‑filtering techniques, Lua scripting, idempotency, capacity planning, monitoring, and compensation strategies to prevent overselling and ensure reliability.

Distributed SystemsJavaKafka
0 likes · 31 min read
High‑Concurrency Flash‑Sale Blueprint: Redis Atomic Stock and Kafka Throttling
Cloud Architecture
Cloud Architecture
May 8, 2026 · Databases

Slow Queries Causing Outages? Build a Cloud‑Native Distributed MySQL Slow‑Log Platform from Scratch

This article walks through the design and implementation of a production‑grade, cloud‑native MySQL slow‑log collection and analysis platform, covering everything from MySQL slow‑log fundamentals and multi‑node ingestion to Kafka buffering, Go‑based parsing, SQL fingerprinting, Elasticsearch and ClickHouse storage, alerting, APM integration, and a phased rollout roadmap.

Cloud NativeKafkaMySQL
0 likes · 35 min read
Slow Queries Causing Outages? Build a Cloud‑Native Distributed MySQL Slow‑Log Platform from Scratch
Cloud Architecture
Cloud Architecture
May 6, 2026 · Backend Development

Why Polling Breaks at Scale and How Spring Boot Webhooks Enable High‑Concurrency Event‑Driven Architecture

The article explains why traditional polling becomes unsustainable under high load, contrasts polling with webhook‑based event delivery, and provides a complete Spring Boot implementation—including outbox pattern, Kafka integration, retry logic, security, observability, and operational best practices—to build a production‑grade, scalable webhook platform.

KafkaOutboxSpring Boot
0 likes · 36 min read
Why Polling Breaks at Scale and How Spring Boot Webhooks Enable High‑Concurrency Event‑Driven Architecture
Lobster Programming
Lobster Programming
May 6, 2026 · Backend Development

How to Choose the Right MQ: RabbitMQ vs RocketMQ vs Kafka

This article compares RabbitMQ, RocketMQ, and Kafka on throughput, latency, scalability, and reliability, outlining each system's core features and recommending suitable scenarios such as reliable messaging, high‑performance streaming, and large‑scale real‑time data processing.

KafkaMessage QueueRabbitMQ
0 likes · 6 min read
How to Choose the Right MQ: RabbitMQ vs RocketMQ vs Kafka
Cloud Architecture
Cloud Architecture
May 3, 2026 · Big Data

Cutting Log Storage Costs 70% for 100 Billion Daily Logs: A Full Guide to Hot‑Cold Separation Architecture

This article explains why massive log systems must adopt hot‑cold separation, walks through the problem analysis, SLO definition, component design, Kafka partition planning, Elasticsearch and ClickHouse tuning, Parquet archiving, a unified query gateway with async cold queries, governance practices, cost modeling, common pitfalls, and a roadmap for evolving the platform.

ElasticsearchKafkaLog Storage
0 likes · 40 min read
Cutting Log Storage Costs 70% for 100 Billion Daily Logs: A Full Guide to Hot‑Cold Separation Architecture
Woodpecker Software Testing
Woodpecker Software Testing
Apr 30, 2026 · Databases

Datafaker: A Powerful Tool for Bulk Test Data Generation

Datafaker is a Python‑compatible utility that creates large volumes of synthetic test data for databases, streams, files, and messaging systems, offering flexible metadata rules, multi‑backend support, and command‑line options for quick data provisioning.

ElasticsearchKafkaMetadata
0 likes · 14 min read
Datafaker: A Powerful Tool for Bulk Test Data Generation
Java Tech Workshop
Java Tech Workshop
Apr 29, 2026 · Backend Development

How to Diagnose and Scale SpringBoot Message Backlog with Monitoring

The article explains why message backlog occurs in SpringBoot applications, outlines systematic troubleshooting steps, proposes comprehensive monitoring across producer, broker, and consumer layers, and presents scaling tactics such as instance expansion, concurrency tuning, batch consumption, and long‑term capacity planning.

BacklogKafkaMessage Queue
0 likes · 16 min read
How to Diagnose and Scale SpringBoot Message Backlog with Monitoring
Java Tech Workshop
Java Tech Workshop
Apr 29, 2026 · Backend Development

Mastering SpringBoot Transactional Messaging for Distributed Consistency

This article explains how SpringBoot handles transactional messaging to achieve distributed data consistency, covering the concept of transaction messages, CAP theory, final consistency, and three practical implementations using a local message table, Kafka transactions, and RocketMQ, plus idempotency and compensation strategies.

CAP theoremDistributed ConsistencyIdempotency
0 likes · 17 min read
Mastering SpringBoot Transactional Messaging for Distributed Consistency
Java Tech Workshop
Java Tech Workshop
Apr 28, 2026 · Backend Development

Ensuring Message Order in SpringBoot: Partitioning and Sequential Consumption

This article examines why message ordering is critical in distributed systems, explains how partition mechanisms in Kafka, RocketMQ, and RabbitMQ enable ordered consumption, and provides detailed SpringBoot implementations, best‑practice guidelines, partition‑key design principles, concurrency settings, idempotency, and real‑world case studies to ensure reliable sequential processing.

Backend DevelopmentKafkaPartitioning
0 likes · 28 min read
Ensuring Message Order in SpringBoot: Partitioning and Sequential Consumption
Golang Shines
Golang Shines
Apr 28, 2026 · Backend Development

Essential Go Packages for Production Environments

This article compiles a curated list of production‑ready Go packages covering testing, logging, error handling, caching, databases, HTTP routing, HTTP clients, fault tolerance, Kafka, and various utility libraries, explaining their key features, concrete code examples, and why they are preferred in real‑world services.

Fault ToleranceGoHTTP
0 likes · 15 min read
Essential Go Packages for Production Environments
LuTiao Programming
LuTiao Programming
Apr 28, 2026 · Backend Development

How I Built a High‑Performance Java Price‑Comparison Engine from Scratch

Starting from a simple sequential Java price‑aggregator, the article walks through successive architectural upgrades—concurrent calls with CompletableFuture, timeout and fallback handling, Spring Boot service exposure, caching, bulkhead isolation, microservice split, and Kafka‑driven event processing—showing how latency drops from 1500 ms to under 20 ms.

ConcurrencyJavaKafka
0 likes · 9 min read
How I Built a High‑Performance Java Price‑Comparison Engine from Scratch
Java Tech Workshop
Java Tech Workshop
Apr 27, 2026 · Backend Development

How to Integrate Kafka with SpringBoot for High‑Performance Messaging

This article walks through Kafka’s core architecture, explains why it achieves massive throughput, and provides a step‑by‑step SpringBoot integration—including environment setup, Maven dependencies, configuration, producer and consumer code, advanced features like transactions and dead‑letter queues, plus performance monitoring and tuning tips.

ConsumerDead Letter QueueJava
0 likes · 11 min read
How to Integrate Kafka with SpringBoot for High‑Performance Messaging
DevOps Coach
DevOps Coach
Apr 26, 2026 · Backend Development

Forget Kafka: A Lightweight Go Queue Achieves 2 Million Messages per Second

The article analyzes how replacing Kafka with a simple in‑memory Go queue reduced architectural complexity, boosted throughput from 240‑330 K to 1.8‑2.0 M messages per second, and clarified debugging, while still acknowledging scenarios where Kafka remains the better choice.

Backend PerformanceGoIn‑Memory Ring Buffer
0 likes · 8 min read
Forget Kafka: A Lightweight Go Queue Achieves 2 Million Messages per Second
Cloud Architecture
Cloud Architecture
Apr 23, 2026 · Backend Development

Gracefully Stopping Java Threads: Kernel Mechanics to Cloud‑Native Guide

This article explains why Java threads cannot be force‑killed, how Thread.interrupt works, and provides a step‑by‑step framework—including cooperative cancellation, two‑phase termination, thread‑pool shutdown, Kafka consumer handling, Spring Boot lifecycle integration, and Kubernetes termination orchestration—to achieve safe, observable shutdown of production‑grade Java services.

ConcurrencyJavaKafka
0 likes · 31 min read
Gracefully Stopping Java Threads: Kernel Mechanics to Cloud‑Native Guide
LuTiao Programming
LuTiao Programming
Apr 22, 2026 · Backend Development

Kafka Deep Dive: Core Concepts Every Architect Must Master to Prevent Outages

The article explains why merely “knowing how to use” Kafka is insufficient, detailing how offset commits, consumer acknowledgments, producer acks, and rebalance behavior affect reliability, and provides concrete code examples, risk scenarios, and configuration recommendations to prevent message loss and duplicate processing in production systems.

KafkaProducer acksconsumer offset
0 likes · 7 min read
Kafka Deep Dive: Core Concepts Every Architect Must Master to Prevent Outages
ITPUB
ITPUB
Apr 17, 2026 · Industry Insights

Why LinkedIn Dumped Kafka for Its Own ‘Northguard’ Streaming Engine

LinkedIn, the original home of Apache Kafka, abandoned the platform for a home‑grown system called Northguard, redesigning log storage, decentralizing metadata, and adding a virtualized Xinfra layer to handle trillions of daily events, while still acknowledging Kafka’s relevance for most companies.

Distributed SystemsKafkaLinkedIn
0 likes · 7 min read
Why LinkedIn Dumped Kafka for Its Own ‘Northguard’ Streaming Engine
Architect Chen
Architect Chen
Apr 16, 2026 · Big Data

Supercharge Kafka Consumer Performance: Parallelism, Batching, and Multithreading

This guide explains practical techniques to dramatically increase Kafka consumer throughput, including scaling consumer instances or partitions, tuning fetch and poll parameters, and implementing a multithreaded consumer model, while also covering hardware, JVM, and OS optimizations and monitoring recommendations.

Batch FetchKafkaMultithreading
0 likes · 5 min read
Supercharge Kafka Consumer Performance: Parallelism, Batching, and Multithreading
Cloud Architecture
Cloud Architecture
Apr 13, 2026 · Operations

Ultimate Message Middleware Comparison: Choosing Kafka, RabbitMQ or RocketMQ

This article walks through why modern systems need a message broker, presents a four‑layer architectural view, compares Kafka, RabbitMQ and RocketMQ across design, performance, routing and transactional capabilities, and offers concrete scenario recommendations, engineering best‑practices, code samples and monitoring guidelines.

KafkaMessage QueuePerformance
0 likes · 33 min read
Ultimate Message Middleware Comparison: Choosing Kafka, RabbitMQ or RocketMQ
LuTiao Programming
LuTiao Programming
Apr 8, 2026 · Backend Development

From Chaos to Production: Building a Real Food-Delivery Backend with Spring Boot

The article chronicles the step‑by‑step evolution of a small team’s chaotic food‑delivery backend into a production‑ready system, detailing how they introduced layering, transactions, security, caching, async processing, messaging, observability, scalability, resilience, containerization, and testing using Spring Boot, Kafka, Redis, JWT, and Docker.

Backend ArchitectureDockerKafka
0 likes · 10 min read
From Chaos to Production: Building a Real Food-Delivery Backend with Spring Boot
Ray's Galactic Tech
Ray's Galactic Tech
Apr 4, 2026 · Backend Development

How to Build a High‑Concurrency Story Creation Platform with AgentScope Java

This article presents a step‑by‑step engineering guide for constructing a production‑grade, high‑throughput story generation platform using AgentScope Java, Spring Boot, Kafka, Redis, PostgreSQL, and Kubernetes, covering architecture, task modeling, DAG orchestration, code organization, scalability, observability, and deployment best practices.

JavaKafkaSpring Boot
0 likes · 39 min read
How to Build a High‑Concurrency Story Creation Platform with AgentScope Java
Cloud Architecture
Cloud Architecture
Apr 3, 2026 · Operations

Zero‑Downtime High‑Throughput Kafka MirrorMaker 2 Migration with Dual‑Write and Rollback

This guide details a production‑grade Kafka MirrorMaker 2 migration for an e‑commerce order platform, covering zero‑downtime strategies, high‑throughput dual‑write, capacity planning, Kubernetes deployment, monitoring, offset continuity, gray‑scale cut‑over, rollback procedures, and extensive configuration examples.

DualWriteKafkaMigration
0 likes · 34 min read
Zero‑Downtime High‑Throughput Kafka MirrorMaker 2 Migration with Dual‑Write and Rollback
Alibaba Cloud Native
Alibaba Cloud Native
Mar 30, 2026 · Industry Insights

How Haier’s AIoT Platform Scaled to Billions of Messages with Kafka Serverless on Alibaba Cloud

The article details how Haier Smart Home’s AIoT platform tackled massive device messaging demands by migrating its self‑built Kafka clusters to Alibaba Cloud’s Kafka Serverless, outlining the technical challenges, step‑by‑step migration plan, custom performance tuning, risk‑co‑governance, and the resulting improvements in stability, throughput, and operational efficiency.

AIoTAlibaba CloudCloud Migration
0 likes · 11 min read
How Haier’s AIoT Platform Scaled to Billions of Messages with Kafka Serverless on Alibaba Cloud
Cloud Architecture
Cloud Architecture
Mar 29, 2026 · Backend Development

Smart Parking Guidance Using Kafka, Flink, and Spring Boot

This article presents a senior architect’s end‑to‑end design of a smart parking guidance system, detailing how high‑frequency sensor streams are ingested via Spring Boot gateways, processed with Kafka and Flink for stateful de‑duplication, debouncing, and aggregation, and finally served through Redis, PostgreSQL and Spring Boot APIs for real‑time vehicle routing.

FlinkKafkaSmart Parking
0 likes · 35 min read
Smart Parking Guidance Using Kafka, Flink, and Spring Boot
Cloud Architecture
Cloud Architecture
Mar 29, 2026 · Backend Development

Kafka Ordered Writes and Reads: The Full Truth Behind Message Sequencing

This article demystifies Kafka's ordering guarantees, explains the difference between partition‑level, key‑level, and global ordering, and provides a complete, production‑grade guide—including architecture, configuration, code samples, and best‑practice patterns—to achieve reliable ordered processing while maintaining high throughput and scalability.

CDCConsumerKafka
0 likes · 35 min read
Kafka Ordered Writes and Reads: The Full Truth Behind Message Sequencing
360 Zhihui Cloud Developer
360 Zhihui Cloud Developer
Mar 27, 2026 · Cloud Native

How AutoMQ Transforms Kafka into a Cloud‑Native, Elastic Messaging Service

This article examines the limitations of traditional Kafka in large‑scale deployments and presents AutoMQ’s cloud‑native redesign—detailing its stateless architecture, storage separation, automatic scaling, read/write isolation, performance benchmarks, and real‑world migration case studies that demonstrate reduced latency, higher throughput, and lower resource costs.

AutoMQCloud NativeElastic Scaling
0 likes · 13 min read
How AutoMQ Transforms Kafka into a Cloud‑Native, Elastic Messaging Service
Cloud Architecture
Cloud Architecture
Mar 22, 2026 · Backend Development

Message Queues Aren’t a Silver Bullet: Decoupling, Asynchrony & Peak‑Shaving

The article analyses why message queues are not a cure‑all, using real e‑commerce and finance incidents to dissect the three core benefits—decoupling, asynchronous processing, and peak‑shaving—while exposing production challenges such as loss, duplication, ordering, dead‑letter handling, and end‑to‑end consistency, and offering concrete Java code, benchmark tables and a practical MQ selection guide.

DecouplingDistributed SystemsKafka
0 likes · 32 min read
Message Queues Aren’t a Silver Bullet: Decoupling, Asynchrony & Peak‑Shaving
Alibaba Cloud Native
Alibaba Cloud Native
Mar 20, 2026 · Cloud Native

How a Gaming Platform Scaled to Millions with RocketMQ & Kafka: A Cloud‑Native Success Story

Facing explosive growth, the game‑service platform 悠悠有品 rebuilt its architecture on Alibaba Cloud, using RocketMQ for core transaction messaging and Kafka for data synchronization, achieving elastic scaling, high availability, cost reduction, and reliable high‑concurrency processing across its trading and analytics pipelines.

Game PlatformKafkaRocketMQ
0 likes · 8 min read
How a Gaming Platform Scaled to Millions with RocketMQ & Kafka: A Cloud‑Native Success Story
dbaplus Community
dbaplus Community
Mar 5, 2026 · Backend Development

How to Ensure Message Order in Kafka: From Basics to Advanced Solutions

This article explains the concept of message ordering in distributed systems, details how Kafka stores messages in partitions, compares global and partial ordering, evaluates single‑partition, asynchronous, and multi‑partition solutions—including handling data skew and partition expansion—and provides a practical interview guide.

BackendDistributed SystemsKafka
0 likes · 22 min read
How to Ensure Message Order in Kafka: From Basics to Advanced Solutions
Architect Chen
Architect Chen
Mar 3, 2026 · Backend Development

Preventing Kafka Duplicate Consumption with Idempotent Design

This article explains practical strategies to avoid duplicate message consumption in Kafka, covering business idempotency with unique IDs, database or Redis deduplication tables, enabling producer idempotence, consumer-side checks, and Kafka's transaction-based exactly‑once semantics, along with their trade‑offs and suitable scenarios.

BackendIdempotencyKafka
0 likes · 4 min read
Preventing Kafka Duplicate Consumption with Idempotent Design
Golang Shines
Golang Shines
Feb 25, 2026 · Big Data

Essential Kafka Core Concepts – A Comprehensive Cheat Sheet (PDF Included)

This article outlines Kafka's role as a leading MQ middleware, highlighting its key traits of decoupling, asynchronous processing, and throttling, compares it with traditional messaging systems, and provides a structured list of basic, intermediate, and advanced topics plus interview questions for in‑depth study.

KafkaMessage Queuedata integration
0 likes · 5 min read
Essential Kafka Core Concepts – A Comprehensive Cheat Sheet (PDF Included)
ITPUB
ITPUB
Feb 11, 2026 · Backend Development

How to Guarantee Zero Message Loss in MQ Systems: A Full‑Lifecycle Design

This guide explains why guaranteeing 100% message reliability in MQ is a critical system‑design interview topic and presents a three‑layer architecture—production, storage, and consumption—detailing ACK settings, local message tables, broker replication, leader election safeguards, manual offset commits, and idempotent processing to prevent any message loss.

Distributed SystemsIdempotencyKafka
0 likes · 11 min read
How to Guarantee Zero Message Loss in MQ Systems: A Full‑Lifecycle Design
ITPUB
ITPUB
Feb 9, 2026 · Databases

ClickHouse vs Doris vs Redis: Real‑World Query Performance Test with Flink

Using a 600k‑record IP range dataset, we built identical tables in ClickHouse and Doris, and a Redis skip‑list store, then ran three Flink‑Kafka streaming jobs to compare query latency across the three databases under varying traffic rates, revealing Redis as fastest, ClickHouse second, Doris slowest.

ClickHouseDatabase PerformanceDoris
0 likes · 8 min read
ClickHouse vs Doris vs Redis: Real‑World Query Performance Test with Flink
Top Architect
Top Architect
Feb 4, 2026 · Backend Development

Build a Robust Asynchronous Processing SDK with Spring, Kafka and MySQL

This article introduces a generic asynchronous processing SDK for Java back‑ends, explaining its design principles, advantages, component architecture, database schema, configuration via Apollo, usage steps, and practical demonstrations, while providing complete code snippets and a GitHub repository for reference.

JavaKafkaSpring
0 likes · 12 min read
Build a Robust Asynchronous Processing SDK with Spring, Kafka and MySQL
Tencent Cloud Developer
Tencent Cloud Developer
Feb 4, 2026 · Backend Development

How We Cut Server Costs by 82%: Refactoring a High‑Concurrency QQ Game Service from C++ to Go with Kafka

This article details the redesign of a core QQ game achievement service that suffered from low resource utilization and heavy CAS contention, describing how moving from a synchronous C++ implementation to an asynchronous Go‑Kafka pipeline eliminated lock conflicts, reduced server count by 82%, and dramatically improved latency and stability.

Kafkahigh-concurrencyrefactoring
0 likes · 11 min read
How We Cut Server Costs by 82%: Refactoring a High‑Concurrency QQ Game Service from C++ to Go with Kafka
Java Tech Enthusiast
Java Tech Enthusiast
Feb 3, 2026 · Backend Development

Spring Boot 4.0.2: Critical Kafka Fixes, Dependency Upgrades & What to Watch

Spring Boot 4.0.2 is a maintenance release that focuses on fixing over 20 bugs—including a critical Kafka transaction auto‑configuration issue—upgrading more than 40 core dependencies, improving documentation, and introducing minor breaking changes, making it essential for Java developers to upgrade promptly.

Kafkabug fixdependency-upgrade
0 likes · 8 min read
Spring Boot 4.0.2: Critical Kafka Fixes, Dependency Upgrades & What to Watch
LuTiao Programming
LuTiao Programming
Jan 30, 2026 · Backend Development

Spring Boot 4.0.2 Released: Over 20 Critical Bugs Fixed and Kafka Transaction Issue Resolved

Spring Boot 4.0.2, launched on January 22, brings more than 20 critical bug fixes, restores Kafka transaction auto‑configuration, improves startup performance by up to 15%, stabilizes native‑image support, refines test behavior, upgrades core dependencies, and introduces a few breaking changes, making it a must‑upgrade for production users.

KafkaNative ImagePerformance
0 likes · 8 min read
Spring Boot 4.0.2 Released: Over 20 Critical Bugs Fixed and Kafka Transaction Issue Resolved
Ray's Galactic Tech
Ray's Galactic Tech
Jan 30, 2026 · Cloud Native

Scale a Monolithic Article Interaction Service with Kubernetes Microservices

This article walks through converting a single‑service article interaction module—handling likes, favorites, and reads—into independent microservices deployed on Kubernetes, detailing architecture goals, service separation, Redis‑based high‑concurrency handling, Kafka async persistence, deployment configurations, auto‑scaling, and real‑world performance results.

KafkaRedis
0 likes · 8 min read
Scale a Monolithic Article Interaction Service with Kubernetes Microservices
Su San Talks Tech
Su San Talks Tech
Jan 30, 2026 · Backend Development

Why Kafka Rebalance Causes Backlog, Duplicates, and Data Loss—and How to Fix It

Kafka consumer group rebalances can trigger message backlogs, duplicate processing, and data loss; this article explains common rebalance triggers, their impact on consumption, and practical configuration and coding strategies—such as tuning timeout parameters, using manual offset commits, and sticky partition assignment—to minimize disruptions.

Consumer GroupKafkaMessage Duplication
0 likes · 12 min read
Why Kafka Rebalance Causes Backlog, Duplicates, and Data Loss—and How to Fix It
LuTiao Programming
LuTiao Programming
Jan 27, 2026 · Big Data

Why LinkedIn Is Replacing Kafka with Its Own Next‑Gen Streaming System

LinkedIn, facing planetary‑scale data volumes, found Kafka’s architecture hitting fundamental limits and built Northguard—a decentralized, log‑striped streaming platform with Raft‑based metadata and an Xinfra migration layer—to gradually replace Kafka’s core responsibilities while maintaining compatibility.

Distributed SystemsKafkaLinkedIn
0 likes · 8 min read
Why LinkedIn Is Replacing Kafka with Its Own Next‑Gen Streaming System
DevOps Coach
DevOps Coach
Jan 27, 2026 · Backend Development

7 Essential Kafka Design Patterns Every Engineer Should Master

This guide presents seven practical Kafka design patterns—single‑key single‑write, log compaction, multi‑consumer‑group fan‑out, retry and dead‑letter topics, exactly‑once processing with Streams, schema evolution with Avro, and choreography vs orchestration—detailing when to use each, core principles, code examples, tips, common pitfalls, and final recommendations for building reliable, observable, and maintainable event‑driven systems.

Kafkadesign patternsevent streaming
0 likes · 9 min read
7 Essential Kafka Design Patterns Every Engineer Should Master
Ray's Galactic Tech
Ray's Galactic Tech
Jan 23, 2026 · Backend Development

How to Build a Kafka‑Level High‑Performance Message Queue from Scratch

This article presents a step‑by‑step guide to designing and implementing a Kafka‑class distributed log‑based message queue kernel, covering architecture, sequential writes, sparse indexing, zero‑copy I/O, partitioning, replication, consumer‑group metadata, batch pipelines, crash recovery, and performance benchmarks.

KafkaLog StorageMessage Queue
0 likes · 7 min read
How to Build a Kafka‑Level High‑Performance Message Queue from Scratch
Architect's Guide
Architect's Guide
Jan 22, 2026 · Big Data

Unlock Kafka’s Power: Core Concepts, High‑Performance Architecture & Real‑World Scaling Tips

This comprehensive guide explores Kafka’s core value as a message queue, explains producers, consumers, topics, partitions, and replication, dives into cluster architecture, zero‑copy I/O, resource planning for disks, memory, CPU and network, and provides practical configuration, consumer‑group management, and operational tooling tips for building high‑throughput, highly available Kafka deployments.

Cluster ScalingDistributed SystemsKafka
0 likes · 31 min read
Unlock Kafka’s Power: Core Concepts, High‑Performance Architecture & Real‑World Scaling Tips
ITPUB
ITPUB
Jan 21, 2026 · Interview Experience

How to Design a Billion‑User Real‑Time Step Leaderboard for Interviews

This article breaks down the interview‑level system design of a WeChat‑style step leaderboard that must support over a billion users, handling massive write spikes, low‑latency friend ranking queries, storage scaling, and relationship complexity with a three‑part architecture using MQ, Redis, and MySQL.

KafkaLeaderboardRedis
0 likes · 8 min read
How to Design a Billion‑User Real‑Time Step Leaderboard for Interviews
Tech Freedom Circle
Tech Freedom Circle
Jan 15, 2026 · Backend Development

Kafka Rebalance Storm Crushed 120k QPS in JD Interview – How to Understand and Fix

In a JD senior Java architect interview, a Kafka consumer‑group rebalance storm caused QPS to drop from 120k to zero, triggering massive message loss and latency spikes, and the article walks through the rebalance fundamentals, failure causes, impact analysis, cooperative sticky assignor migration, and comprehensive monitoring and mitigation strategies.

Consumer GroupDistributed SystemsKafka
0 likes · 28 min read
Kafka Rebalance Storm Crushed 120k QPS in JD Interview – How to Understand and Fix
Code Wrench
Code Wrench
Jan 14, 2026 · Backend Development

When to Choose NATS Over Kafka for Go Microservices: A Practical Guide

This article compares Kafka, RabbitMQ, and NATS for Go microservices, explains why Kafka is often over‑engineered for internal communication, and shows how NATS provides a lightweight, event‑driven alternative with concrete code examples and a clear selection matrix.

GoKafkaMessage Queue
0 likes · 9 min read
When to Choose NATS Over Kafka for Go Microservices: A Practical Guide
JD Retail Technology
JD Retail Technology
Jan 13, 2026 · Backend Development

Deep Dive into Kafka, RocketMQ, and JMQ Storage Architectures

This article compares the storage models, data organization, indexing, read/write processes, and performance trade‑offs of three major message queues—Kafka, RocketMQ, and JMQ—providing detailed technical insights for architects and engineers making storage‑related design decisions.

Backend EngineeringJMQKafka
0 likes · 16 min read
Deep Dive into Kafka, RocketMQ, and JMQ Storage Architectures
Top Architect
Top Architect
Jan 12, 2026 · Backend Development

How to Build a Robust Asynchronous Processing SDK with Spring, Kafka, and XXL‑Job

This article explains the design and implementation of a generic asynchronous processing SDK for Java, covering its purpose, advantages, core principles, component choices, design patterns, configuration via Apollo, usage steps, safety considerations, and provides complete SQL and Spring configuration examples along with a GitHub repository link.

JavaKafkaSDK
0 likes · 11 min read
How to Build a Robust Asynchronous Processing SDK with Spring, Kafka, and XXL‑Job
iQIYI Technical Product Team
iQIYI Technical Product Team
Jan 8, 2026 · Big Data

How iQIYI Cut Stream Data Costs by 70%: From Private‑Cloud Kafka to AutoMQ

This article details iQIYI's evolution from a tightly coupled private‑cloud Kafka setup to a cloud‑native AutoMQ architecture, describing the challenges of scaling, the development of the Stream platform and Stream‑SDK, the migration to hybrid and public‑cloud Kafka, and the resulting cost and elasticity improvements.

AutoMQCloud MigrationKafka
0 likes · 12 min read
How iQIYI Cut Stream Data Costs by 70%: From Private‑Cloud Kafka to AutoMQ
Ray's Galactic Tech
Ray's Galactic Tech
Jan 6, 2026 · Backend Development

Scalable GPS Data Backend: SpringBoot, Kafka, MongoDB & Redis Design

This guide outlines a complete backend architecture for high‑volume GPS data, detailing the overall system flow, technology stack choices, Maven dependencies, data models, Kafka producer/consumer configurations, SpringBoot controllers, asynchronous processing, Redis caching, health checks, Docker deployment, and performance tuning recommendations to ensure stability and scalability.

DockerKafkaMongoDB
0 likes · 11 min read
Scalable GPS Data Backend: SpringBoot, Kafka, MongoDB & Redis Design
Tech Freedom Circle
Tech Freedom Circle
Jan 6, 2026 · Backend Development

Why Choose RocketMQ Over Kafka? The Real Reasons Behind the 90% Mistake

This article dissects a common interview question about Kafka's higher throughput versus RocketMQ's richer features, explains the underlying design philosophies, storage models, I/O paths, scaling limits, real‑world use cases such as transaction, delayed and ordered messages, and provides concrete optimization steps and code samples to help engineers make an informed messaging platform choice.

Distributed SystemsJavaKafka
0 likes · 42 min read
Why Choose RocketMQ Over Kafka? The Real Reasons Behind the 90% Mistake
JakartaEE China Community
JakartaEE China Community
Jan 6, 2026 · Backend Development

Run a WildFly Application with JBang in Minutes

This tutorial shows how to use JBang scripts to launch a WildFly server, create a simple Jakarta REST endpoint, and extend the server with WildFly Glow add‑ons such as Kafka, providing step‑by‑step commands, code examples, and Docker integration for rapid prototyping.

JBangJakarta EEJava scripting
0 likes · 9 min read
Run a WildFly Application with JBang in Minutes
Architect Chen
Architect Chen
Jan 2, 2026 · Backend Development

Preventing Duplicate Consumption in Kafka: Design, Idempotence, and Configuration Strategies

This guide explains how to avoid duplicate message consumption in Kafka by designing unique identifiers, implementing consumer-side idempotence with deduplication tables, leveraging Kafka’s transactional features, and establishing system-level safeguards and monitoring to ensure reliable, exactly‑once processing.

Backend DevelopmentKafkaMessage Queue
0 likes · 4 min read
Preventing Duplicate Consumption in Kafka: Design, Idempotence, and Configuration Strategies
dbaplus Community
dbaplus Community
Dec 30, 2025 · Backend Development

How to Tackle Massive Message Queue Backlogs in High‑Traffic Scenarios

During peak traffic like Double‑11, a message queue can accumulate millions of messages, and simply adding consumer instances only offers temporary relief; this article explains the partition model limits, how to calculate proper partition numbers, fast remediation tactics, and deep consumer‑side optimizations for robust, scalable processing.

BacklogKafkaMessage Queue
0 likes · 20 min read
How to Tackle Massive Message Queue Backlogs in High‑Traffic Scenarios
Mike Chen's Internet Architecture
Mike Chen's Internet Architecture
Dec 25, 2025 · Backend Development

How to Resolve Kafka Backlog Under High Load: Practical Tips

This article explains why Kafka experiences message backlog in high‑load environments, identifies producer‑consumer speed mismatches, I/O and resource bottlenecks, and offers concrete strategies such as scaling consumers, tuning hardware, and adjusting Kafka configurations to eliminate the backlog.

BacklogDistributed SystemsKafka
0 likes · 4 min read
How to Resolve Kafka Backlog Under High Load: Practical Tips
Mike Chen's Internet Architecture
Mike Chen's Internet Architecture
Dec 22, 2025 · Operations

How to Diagnose and Resolve Kafka Consumer Lag Quickly

When Kafka consumers fall behind, this guide walks you through confirming the backlog, pinpointing bottlenecks in production, consumption, or brokers, and applying concrete steps—such as checking offsets, comparing TPS, inspecting consumer logic, and adjusting partitions—to efficiently eliminate lag.

Consumer LagKafka
0 likes · 4 min read
How to Diagnose and Resolve Kafka Consumer Lag Quickly
Architect Chen
Architect Chen
Dec 21, 2025 · Backend Development

How to Resolve Kafka Backlog: Boost Consumer Throughput and Optimize Partitions

This guide explains why Kafka backlog occurs when production outpaces consumption and provides practical steps—such as increasing consumer instances, optimizing processing, expanding partitions, applying flow‑control, and managing message capacity—to eliminate the backlog and keep the cluster healthy.

BacklogKafkaconsumer scaling
0 likes · 4 min read
How to Resolve Kafka Backlog: Boost Consumer Throughput and Optimize Partitions
Mike Chen's Internet Architecture
Mike Chen's Internet Architecture
Dec 9, 2025 · Backend Development

Boost Kafka to Over 1 Million Messages per Second: Metrics and Tuning Tips

This article explains what high concurrency means for Kafka, outlines key performance metrics such as QPS, TPS, throughput and latency, and provides concrete configuration and architectural techniques—including broker optimization, horizontal scaling, network batching, and zero‑copy—to achieve write rates exceeding one million records per second.

BackendDistributed SystemsKafka
0 likes · 4 min read
Boost Kafka to Over 1 Million Messages per Second: Metrics and Tuning Tips
IT Services Circle
IT Services Circle
Dec 9, 2025 · Backend Development

Mastering Kafka Rebalance: Prevent Backlog, Duplicates, and Data Loss

When Kafka consumer groups rebalance, partitions are reassigned, often causing message backlog, duplicate processing, or loss; understanding the triggers, impact, and optimization techniques—like tuning timeouts, managing offset commits, and using sticky assignors—can keep your streaming pipelines reliable.

Consumer GroupKafkaMessage Backlog
0 likes · 13 min read
Mastering Kafka Rebalance: Prevent Backlog, Duplicates, and Data Loss
dbaplus Community
dbaplus Community
Dec 8, 2025 · Databases

Which Database Wins IP Range Lookups? ClickHouse vs Doris vs Redis Benchmarks

This article presents a systematic benchmark comparing ClickHouse, Doris, and Redis for IP‑range dimension lookups using Flink‑Kafka pipelines, detailing test design, result table schema, query interfaces, and performance results across varying data rates, concluding that Redis offers the fastest and most stable query latency.

ClickHouseDatabase BenchmarkDoris
0 likes · 7 min read
Which Database Wins IP Range Lookups? ClickHouse vs Doris vs Redis Benchmarks