Skip to content
Tagged articles

Kafka

500 articles · Page 1 of 5
Mike Chen's Internet Architecture
Mike Chen's Internet Architecture
Nov 18, 2026 · Backend Development

Message Queues Explained: Principles, Architecture & Selection Guide

This article comprehensively explains message queue fundamentals including decoupling, async communication, reliability, and buffering; details architecture roles, queue types, persistence, acknowledgment, and protocols; compares ActiveMQ, RabbitMQ, Kafka, RocketMQ, and Pulsar with performance metrics; and provides selection guidance for e-commerce, finance, and big data scenarios.

ActiveMQKafkaMQ
0 likes · 8 min read
Message Queues Explained: Principles, Architecture & Selection Guide
Architecture Pony
Architecture Pony
Nov 16, 2026 · Operations

Kafka Production Playbook: 6-Phase Deployment, Capacity Formulas & 10 Incident Runbooks

This article provides a comprehensive Kafka production deployment guide covering a six-phase rollout with mandatory checkpoints, three capacity planning formulas with examples, topic partitioning strategies, security and quota configurations, ten incident categories with symptom/mitigation/root-cause tables, major promotion assurance timeline, chaos engineering practices, and when to choose alternatives like RocketMQ or Pulsar.

Chaos EngineeringKafkaPerformance Testing
0 likes · 42 min read
Kafka Production Playbook: 6-Phase Deployment, Capacity Formulas & 10 Incident Runbooks
Architecture Pony
Architecture Pony
Nov 16, 2026 · Backend Development

Kafka Exactly-Once: Idempotent Producers, Transactions & Two-Phase Commit Explained

This article explains Kafka's exactly-once semantics through idempotent producers, transactions, and read-committed isolation, detailing real incidents, configuration patterns, source-code mechanisms, and the critical boundary that Kafka transactions cannot cover external side effects, requiring application-level idempotence for true end-to-end exactly-once.

Distributed SystemsKafkaexactly-once
0 likes · 45 min read
Kafka Exactly-Once: Idempotent Producers, Transactions & Two-Phase Commit Explained
Architecture Pony
Architecture Pony
Nov 15, 2026 · Backend Development

Kafka Performance Tuning: Systematic Methodology for Bottleneck Detection & Parameter Tuning

This article presents a systematic Kafka performance tuning methodology through real incident analysis, covering write/read paths, zero-copy, batching, compression, broker thread model, partition sizing, consumer tuning, production configurations, benchmarking practices, monitoring metrics, capacity formulas, and source-code insights — emphasizing bottleneck detection before parameter changes.

Kafkabatchingcompression
0 likes · 42 min read
Kafka Performance Tuning: Systematic Methodology for Bottleneck Detection & Parameter Tuning
Architecture Pony
Architecture Pony
Nov 15, 2026 · Backend Development

Kafka Controller & KRaft: Why Metadata Migration Slashes Failover from 5 Minutes to 4 Seconds

This article analyzes Kafka's Controller architecture evolution from ZooKeeper to KRaft, detailing how metadata migration to an internal Raft-replicated log transforms failover from minutes-long rebuilds to seconds-long handoffs, covering deployment models, migration pitfalls, source-code mechanisms, and production monitoring strategies.

ControllerDistributed SystemsFailover
0 likes · 50 min read
Kafka Controller & KRaft: Why Metadata Migration Slashes Failover from 5 Minutes to 4 Seconds
Architecture Pony
Architecture Pony
Nov 15, 2026 · Backend Development

Kafka Replication: ISR, High Watermark, and the Hidden Dangers of acks=all

This article explains Kafka's replication mechanism through ISR and high watermark, revealing why acks=all waits only for in-sync replicas, how the 30-second shrink window works, why min.insync.replicas is essential, and the trade-offs of unclean leader election, with production configurations, source code insights, and incident case studies.

Distributed SystemsHigh WatermarkISR
0 likes · 53 min read
Kafka Replication: ISR, High Watermark, and the Hidden Dangers of acks=all
Architecture Pony
Architecture Pony
Nov 15, 2026 · Backend Development

Kafka Storage Deep Dive: Append-Only Logs, Sparse Indexes & Compact Traps

This article dissects Kafka's on-disk storage architecture, explaining partition segments, sparse indexes, page cache reliance, flush policies, delete vs. compact cleanup, and real-world incidents like index rebuild storms and misconfigured retention, with production configs and capacity formulas.

Kafkacapacity planningconfiguration
0 likes · 50 min read
Kafka Storage Deep Dive: Append-Only Logs, Sparse Indexes & Compact Traps
java1234
java1234
Nov 15, 2026 · Backend Development

FlowMQ Kafka: Decoupled Storage Cuts Costs 94% and Enables Seamless Scaling

FlowMQ Kafka separates compute and storage with S3 and RLog backends, delivering 94% cost savings on cloud object storage and eliminating manual rebalancing through segment-level replication and Flexible Quorum, outperforming Kafka and Redpanda in multi-AZ failure recovery.

Flexible QuorumFlowMQKafka
0 likes · 15 min read
FlowMQ Kafka: Decoupled Storage Cuts Costs 94% and Enables Seamless Scaling
Architect's Guide
Architect's Guide
Nov 11, 2026 · Backend Development

6 Spring Event Best Practices from Production Failures

The author shares six critical Spring Event best practices derived from production incidents, covering graceful shutdown requirements, startup event loss prevention, suitability for eventual consistency over strong consistency, reliability through retry mechanisms, and mandatory idempotency for subscribers.

IdempotencyKafkaSpring Event
0 likes · 11 min read
6 Spring Event Best Practices from Production Failures
Architecture Pony
Architecture Pony
Nov 6, 2026 · Backend Development

Spring Cloud Stream: Binder Abstraction, Partitioning, and Config Pitfalls

This article analyzes Spring Cloud Stream's binder abstraction, partitioning mechanics, and configuration pitfalls through real incidents, source code insights, and production guidelines, revealing why 'swap MQ without code changes' only half-works and how to avoid silent failures in message-driven systems.

Binder AbstractionConfiguration PitfallsKafka
0 likes · 46 min read
Spring Cloud Stream: Binder Abstraction, Partitioning, and Config Pitfalls
Java Tech Workshop
Java Tech Workshop
Nov 4, 2026 · Backend Development

Kafka Rebalance Deep Dive: Eager vs Cooperative Protocols and Eliminating Partition Transfer Pauses

This article analyzes Kafka's consumer group rebalance mechanisms, comparing eager and cooperative protocols, explaining why partition transfers cause consumption pauses, detailing configuration parameters like session.timeout.ms and max.poll.interval.ms, and providing troubleshooting steps for lag spikes during rolling deployments.

Configuration TuningConsumer GroupCooperative Protocol
0 likes · 22 min read
Kafka Rebalance Deep Dive: Eager vs Cooperative Protocols and Eliminating Partition Transfer Pauses
Xiaolin Talks Programming
Xiaolin Talks Programming
Nov 1, 2026 · Backend Development

Building an Enterprise Alert Center with Spring Boot: Convergence, Silencing, On-Call & Escalation

This article details a Spring Boot-based enterprise alert center architecture covering alert ingestion, standardization, four-layer convergence (deduplication, grouping, inhibition, silencing), routing, multi-channel notification, on-call scheduling, escalation policies, and closed-loop governance with SLO integration.

Alert ConvergenceKafkaPrometheus
0 likes · 38 min read
Building an Enterprise Alert Center with Spring Boot: Convergence, Silencing, On-Call & Escalation
Xiaolin Talks Programming
Xiaolin Talks Programming
Oct 28, 2026 · Backend Development

Building a WebRTC Signaling Service with Spring Boot: Room Management, SDP Exchange, Recording Callbacks & Horizontal Scaling

This article details the architecture and implementation of a WebRTC signaling service using Spring Boot 3.2 and WebFlux, covering protocol design, room state management, media server integration, recording callbacks, horizontal scaling strategies, and fault tolerance for real-time audio/video applications.

Horizontal ScalingICEKafka
0 likes · 45 min read
Building a WebRTC Signaling Service with Spring Boot: Room Management, SDP Exchange, Recording Callbacks & Horizontal Scaling
Lakehouse Research Base
Lakehouse Research Base
Oct 18, 2026 · Big Data

Migrating Core Real-Time Indexing from Kafka to Apache Fluss: Xiaohongshu's Engineering Journey

This article details Xiaohongshu's migration of its core real-time indexing pipeline from Kafka to Apache Fluss, covering architectural drivers like read amplification and historical replay contention, Fluss's columnar storage and Lakestream integration with Paimon, and a phased migration approach including baseline testing, shadow dual-run, and gradual cutover.

Apache FlussApache PaimonKafka
0 likes · 50 min read
Migrating Core Real-Time Indexing from Kafka to Apache Fluss: Xiaohongshu's Engineering Journey
Cloud Architecture
Cloud Architecture
Oct 17, 2026 · Backend Development

Designing Go WeChat Mini Program Login for 50k QPS: Distributed Sessions & Async Peak Shaving

This article details a high-throughput WeChat Mini Program login architecture in Go targeting 50k entry QPS, covering distributed session design, async peak shaving with Kafka, capacity modeling using Little's Law, API contracts for sync/async flows, idempotency, transactional outbox, token management, observability, and load testing strategies.

Async Peak ShavingDistributed SessionsGo
0 likes · 39 min read
Designing Go WeChat Mini Program Login for 50k QPS: Distributed Sessions & Async Peak Shaving
Cloud Architecture
Cloud Architecture
Oct 14, 2026 · Backend Development

Go Error Handling & Observability: From Error Trees to Microservice Fault Validation

This article presents a comprehensive Go error handling framework that separates domain causes, protocol results, internal causes, operation context, and execution decisions, demonstrating layered error conversion, deterministic boundary classification, safe HTTP/gRPC responses, cross-service error propagation, observability integration, Kafka offset management, and fault injection validation for production microservices.

Error HandlingGoKafka
0 likes · 32 min read
Go Error Handling & Observability: From Error Trees to Microservice Fault Validation
Spring Full-Stack Practical Cases
Spring Full-Stack Practical Cases
Oct 14, 2026 · Backend Development

Spring Boot Large Batch Request Body? 6 Optimization Strategies

This article presents six optimization strategies for handling oversized batch request bodies in Spring Boot, covering network compression with GZIP, Protobuf adoption, streaming JSON parsing with Jackson JsonParser, early validation layers, asynchronous processing via Kafka, and request chunking to reduce memory pressure and improve throughput.

Batch ProcessingChunked RequestsGZIP Compression
0 likes · 12 min read
Spring Boot Large Batch Request Body? 6 Optimization Strategies
Top Architect
Top Architect
Oct 12, 2026 · Backend Development

Universal Async Processing SDK: Spring Transaction Events + Kafka + XXL-Job for Guaranteed Execution

The article presents fc-async, an open-source universal asynchronous processing SDK that uses Spring's @TransactionalEventListener with fallbackExecution to decouple async tasks from business transactions, persists requests in MySQL, retries via XXL-Job, and consumes via Kafka, achieving eventual consistency without invading core code.

KafkaMySQLSpring
0 likes · 12 min read
Universal Async Processing SDK: Spring Transaction Events + Kafka + XXL-Job for Guaranteed Execution
Full-Stack Internet Architecture
Full-Stack Internet Architecture
Oct 8, 2026 · Backend Development

What Is Apache Kafka? Core Features, Use Cases & Why Engineers Learn It

This article introduces Apache Kafka as a distributed streaming platform, detailing its core advantages—high throughput via sequential writes and zero-copy, horizontal scalability through partitions, durability with 7-day retention and replication—and key use cases including log aggregation, event-driven architectures, real-time analytics, and big data integration with tools like Flink and Spark Streaming.

Data PipelineKafkaKafka Streams
0 likes · 5 min read
What Is Apache Kafka? Core Features, Use Cases & Why Engineers Learn It
Lakehouse Research Base
Lakehouse Research Base
Oct 3, 2026 · Big Data

Lakehouse Capacity Planning: Stress Testing, Bottleneck Detection & Scaling Decisions

This article presents a complete methodology for lakehouse performance stress testing and capacity planning, covering TPC-DS benchmark adaptation, end-to-end bottleneck identification across CPU, memory, disk, and network, a quantitative capacity derivation framework, and vertical versus horizontal scaling decision criteria for components like Kafka, Flink, Paimon, and StarRocks.

FlinkHorizontal ScalingKafka
0 likes · 27 min read
Lakehouse Capacity Planning: Stress Testing, Bottleneck Detection & Scaling Decisions
Ray's Galactic Tech
Ray's Galactic Tech
Sep 29, 2026 · Backend Development

Production-Grade LLM Annotation Pipeline: Pydantic, PostgreSQL & Kafka Patterns

The article details how a SaaS team evolved a fragile LLM ticket-classification script into a production-grade pipeline using strict Pydantic contracts, PostgreSQL state machines with atomic leasing, Kafka outbox patterns, immutable training snapshots, and metric-gated rollouts to handle model drift, replay, and human review safely.

Data EngineeringIdempotencyKafka
0 likes · 27 min read
Production-Grade LLM Annotation Pipeline: Pydantic, PostgreSQL & Kafka Patterns
Niu Liu
Niu Liu
Sep 25, 2026 · Backend Development

Exactly Once: What It Guarantees, Why Payments Double-Charge, and How to Fix It

This article explains that Exactly Once semantics in message queues like Kafka only guarantee exactly-once processing within the messaging system's transaction boundary, not for external side effects like payment APIs, and provides practical patterns using database unique constraints, idempotency keys, and timeout handling to achieve true end-to-end exactly-once behavior.

DeduplicationIdempotencyKafka
0 likes · 9 min read
Exactly Once: What It Guarantees, Why Payments Double-Charge, and How to Fix It
liandk
liandk
Sep 25, 2026 · Backend Development

MQ Production Failure Troubleshooting: Message Loss, Duplicates, Backlog & Dead Letters

This comprehensive guide covers the five critical MQ production failures — message loss, duplicate consumption, massive backlog, consumer hangs, and dead letter queue blocking — with root cause analysis, emergency mitigation steps, and long-term architectural fixes for RocketMQ, Kafka, and RabbitMQ.

Distributed SystemsKafkaMQ
0 likes · 14 min read
MQ Production Failure Troubleshooting: Message Loss, Duplicates, Backlog & Dead Letters
Cloud Architecture
Cloud Architecture
Sep 24, 2026 · Backend Development

Scaling Spring Boot Sign-In to 100M Users: Distributed Architecture Evolution & Production Hardening

This article details the evolution of a Spring Boot daily sign-in system from 10K to 100M users, covering Redis bitmap sharding, Lua atomic operations, command outbox pattern, Kafka async reward processing, idempotency guarantees, fault tolerance strategies, and production-grade observability with real incident postmortems.

BitmapCommand OutboxDistributed Systems
0 likes · 36 min read
Scaling Spring Boot Sign-In to 100M Users: Distributed Architecture Evolution & Production Hardening
Niu Liu
Niu Liu
Sep 23, 2026 · Backend Development

Ordered Messages Aren't Enough: Preventing State Regressions with Versioning and State Machines

The article explains why message ordering alone fails to prevent business state regressions, detailing how version gaps and unordered consumption cause issues, and presents solutions using event versioning, state machines, and proper Kafka/RocketMQ configuration to ensure correct order processing per order ID while allowing parallelism across orders.

Distributed SystemsEvent VersioningKafka
0 likes · 10 min read
Ordered Messages Aren't Enough: Preventing State Regressions with Versioning and State Machines
Niu Liu
Niu Liu
Sep 21, 2026 · Backend Development

Priority Messaging: How Urgent Tasks Cut Ahead Without Starving Regular Tasks

The article analyzes priority messaging in message middleware, comparing implementations across RocketMQ, RabbitMQ, Kafka, and Pulsar, and details techniques like weighted round-robin scheduling, resource reservation, and wait aging to ensure urgent tasks get priority without starving ordinary tasks, including practical code patterns for fair task scheduling.

KafkaPulsarRabbitMQ
0 likes · 10 min read
Priority Messaging: How Urgent Tasks Cut Ahead Without Starving Regular Tasks
Xiaolin Talks Programming
Xiaolin Talks Programming
Sep 5, 2026 · Backend Development

Migrating Log Platform from Elasticsearch to ClickHouse: Table Design, Materialized Views & Tuning Lessons

The author details migrating a high-volume logging platform from Elasticsearch to ClickHouse, covering schema design with ReplicatedMergeTree, Spring Boot integration via JDBC and MyBatis, materialized views for pre-aggregation, and optimization techniques like skip indexes, batch inserts, and capacity planning.

ClickHouseElasticsearchKafka
0 likes · 21 min read
Migrating Log Platform from Elasticsearch to ClickHouse: Table Design, Materialized Views & Tuning Lessons
Spring Full-Stack Practical Cases
Spring Full-Stack Practical Cases
Sep 4, 2026 · Backend Development

10 Message Queue Patterns for Spring Boot: Decoupling, Async, Traffic Shaping & More

This article details 10 practical message queue scenarios for Spring Boot microservices, covering system decoupling, asynchronous processing, traffic shaping for flash sales, data synchronization, centralized logging, broadcast configuration updates, ordered message processing, delayed messages for timeouts, retry mechanisms with dead-letter queues, and transactional messaging for distributed consistency, with code examples for RabbitMQ, Kafka, and RocketMQ.

Asynchronous ProcessingDistributed SystemsKafka
0 likes · 25 min read
10 Message Queue Patterns for Spring Boot: Decoupling, Async, Traffic Shaping & More
LuTiao Programming
LuTiao Programming
Sep 3, 2026 · Backend Development

Kafka 4.2 Share Groups: Can Kafka Replace RabbitMQ for Task Queues?

The author tests Kafka 4.2 Share Groups (Kafka Queues) by migrating a report-generation task queue from RabbitMQ, showing how Share Groups enable elastic consumer scaling beyond partition limits, manual acknowledgment modes (ACCEPT/RELEASE/REJECT/RENEW), and long-task handling with renew(), while noting RabbitMQ's routing strengths remain relevant.

KafkaKafka QueuesMessage Queue
0 likes · 16 min read
Kafka 4.2 Share Groups: Can Kafka Replace RabbitMQ for Task Queues?
Data Integration and Governance
Data Integration and Governance
Sep 1, 2026 · Big Data

Building a Real-Time Data Warehouse: End-to-End Pipeline from Kafka to BI

This article details the end-to-end architecture of a real-time data warehouse, covering data ingestion via CDC and Kafka, stream processing challenges like deduplication and event-time handling, layered modeling (ODS/DWD/DWS/ADS), unified metric definitions, and critical production monitoring for latency, lag, data reconciliation, and consistency with offline results.

CDCFlinkKafka
0 likes · 20 min read
Building a Real-Time Data Warehouse: End-to-End Pipeline from Kafka to BI
Niu Liu
Niu Liu
Aug 31, 2026 · R&D Management

Enterprise Codex Governance: Trustworthy Collection, Secure Access, Auditable Lifecycle

This article details the architecture of an enterprise governance platform for Codex AI coding assistants, covering local encrypted collection with 90ms hooks, server-side parsing with tenant isolation, tiered content access (L0–L3) requiring multi-party approval, and operational use cases from deployment health monitoring to employee offboarding with zero-residue verification.

AI coding assistantCodexDevice Binding
0 likes · 17 min read
Enterprise Codex Governance: Trustworthy Collection, Secure Access, Auditable Lifecycle
Alibaba Cloud Native
Alibaba Cloud Native
Aug 28, 2026 · Industry Insights

When Model Power Is Plenty, Real‑Time Data Pipelines Become the AI Production Bottleneck

As AI models become sufficiently capable, the primary obstacle to production shifts from model selection to the real‑time data link, requiring new load handling, sub‑second context availability, and tighter trust boundaries, prompting a redesign of streaming platforms, consumption models, and security mechanisms.

AICloud NativeEvent-driven AI
0 likes · 16 min read
When Model Power Is Plenty, Real‑Time Data Pipelines Become the AI Production Bottleneck
Woodpecker Software Testing
Woodpecker Software Testing
Aug 28, 2026 · Backend Development

Mastering Concurrent User Testing: Deep Dive into Performance Optimization

The article explains that true concurrent user testing must emulate realistic user sessions with think time and asynchronous actions, identifies three typical bottlenecks—application thread blocking, database connection pool exhaustion, and middleware resource contention—and demonstrates proactive performance‑left‑shift practices and emerging AI‑driven adaptive testing techniques.

AIConcurrent TestingHikariCP
0 likes · 7 min read
Mastering Concurrent User Testing: Deep Dive into Performance Optimization
Cloud Architecture
Cloud Architecture
Aug 27, 2026 · Backend Development

Reliable Delivery for Billion-Scale LongConn Gateways: Tackling ACK Storms

The article explains why a successful Netty write does not guarantee message delivery, and presents a production‑grade solution for billion‑scale long‑connection gateways that combines application‑level ACKs, timeout‑driven retries, offline compensation, state‑machine design, back‑pressure handling, and observability to prevent ACK storms and retry avalanches.

ACK StormBackend DevelopmentKafka
0 likes · 29 min read
Reliable Delivery for Billion-Scale LongConn Gateways: Tackling ACK Storms
Ray's Galactic Tech
Ray's Galactic Tech
Aug 26, 2026 · Big Data

Layered Real‑Time Data Warehouse with Flink CDC, Kafka & Doris

The article explains why real‑time data‑warehouse projects often fail in production and presents a complete, production‑ready solution that layers ODS, DWD, DWS and ADS using Flink CDC to capture MySQL changes, Kafka for buffering and replay, and Doris for OLAP storage, with detailed guidance on architecture, state handling, fault‑tolerance and operations.

CDCData EngineeringDoris
0 likes · 35 min read
Layered Real‑Time Data Warehouse with Flink CDC, Kafka & Doris
Architect Chen
Architect Chen
Aug 21, 2026 · Operations

All Kafka Development Commands Explained – 2026 Edition

This guide walks through the essential Kafka command‑line tools for creating, listing, describing, and altering topics, producing and consuming messages, and managing consumer groups, providing concrete examples and key parameters for each operation.

ConsumerConsumer GroupKafka
0 likes · 5 min read
All Kafka Development Commands Explained – 2026 Edition
Coder Trainee
Coder Trainee
Aug 20, 2026 · Backend Development

How I Resolved a 1‑Million‑Message Kafka Consumer Backlog

When a Kafka topic’s consumer lag suddenly spiked to over one million messages, the author traced the issue to improper acknowledgment handling, implemented emergency scaling, corrected exception processing, added a dead‑letter queue, and created a reusable checklist to prevent future backlogs.

Consumer LagDead Letter QueueKafka
0 likes · 7 min read
How I Resolved a 1‑Million‑Message Kafka Consumer Backlog
Ray's Galactic Tech
Ray's Galactic Tech
Aug 19, 2026 · Backend Development

Designing a 12‑State Payment System State Machine from Pending to Completed

This article presents a complete design for a 12‑state payment order state machine that handles high‑concurrency scenarios such as payment callbacks arriving after an order has been cancelled, using explicit state transitions, database CAS updates, Redis + DB idempotency, Outbox pattern and Kafka‑driven asynchronous processing to achieve reliable, auditable and compensatable order lifecycle management.

ConcurrencyIdempotencyKafka
0 likes · 39 min read
Designing a 12‑State Payment System State Machine from Pending to Completed
Cloud Architecture
Cloud Architecture
Aug 18, 2026 · Backend Development

Scalable Enterprise Real‑Time Push with Spring Boot, WebFlux, Kafka & Redis

This guide walks through why traditional polling or WebSocket solutions quickly break at scale, explains the Server‑Sent Events protocol, and presents a production‑grade architecture that combines Spring Boot 3, WebFlux, Kafka, Redis, and Kubernetes to deliver reliable, ordered, and observable one‑way push notifications for millions of concurrent users.

KafkaRedisSSE
0 likes · 43 min read
Scalable Enterprise Real‑Time Push with Spring Boot, WebFlux, Kafka & Redis
Rain's Path to Java Mastery
Rain's Path to Java Mastery
Aug 17, 2026 · Backend Development

RabbitMQ vs Kafka vs RocketMQ: Complete MQ Selection Guide with Code & Interview Answers

This article provides a comprehensive comparison of RabbitMQ, Kafka, and RocketMQ across core features, performance, and use cases, with production-ready code for transactional messaging, idempotent consumption, and zero-loss configurations, plus solutions for seven common production pitfalls and interview answer templates.

Distributed SystemsIdempotencyKafka
0 likes · 36 min read
RabbitMQ vs Kafka vs RocketMQ: Complete MQ Selection Guide with Code & Interview Answers
Ray's Galactic Tech
Ray's Galactic Tech
Aug 14, 2026 · Backend Development

Payment Callback After Order Cancellation? 3 Consistency Challenges and Practical Engineering Solutions

The article examines why payment callbacks that arrive later than order cancellations cause money‑in‑order‑canceled anomalies, outlines the three core eventual‑consistency problems—message ordering, duplicate consumption, and message loss—and presents a complete engineering solution using business idempotency keys, a transactional outbox, state‑machine handling, and reconciliation compensation.

Distributed SystemsIdempotencyKafka
0 likes · 35 min read
Payment Callback After Order Cancellation? 3 Consistency Challenges and Practical Engineering Solutions
Ray's Galactic Tech
Ray's Galactic Tech
Aug 12, 2026 · Backend Development

Message Queue Showdown: When to Use Kafka, RabbitMQ, or Pulsar

This article dissects common MQ mis‑selections, categorises four message types, explains the five architectural roles of a message broker, compares Kafka, RabbitMQ and Pulsar on capabilities and trade‑offs, and provides a step‑by‑step selection guide with real‑world code snippets and best‑practice patterns.

KafkaMessage QueueOutbox Pattern
0 likes · 26 min read
Message Queue Showdown: When to Use Kafka, RabbitMQ, or Pulsar
Coder Life Journal
Coder Life Journal
Aug 11, 2026 · Backend Development

Spring Cloud + Kafka: 6 Common Pitfalls and How to Avoid Them

The article walks through six real‑world pitfalls when integrating Spring Cloud with Kafka—message loss, duplicate processing, out‑of‑order events, massive consumer lag, serialization mismatches, and misuse of Kafka transactions—and provides concrete configuration tweaks, code examples, and operational safeguards to prevent each issue.

Distributed SystemsKafkaSerialization
0 likes · 9 min read
Spring Cloud + Kafka: 6 Common Pitfalls and How to Avoid Them
ITPUB
ITPUB
Aug 9, 2026 · Big Data

Why a CK+Kafka+Filebeat Stack Beats ELK for Log Analytics

The article compares Elasticsearch and ClickHouse for log storage, analyzes cost and performance, and provides a step‑by‑step guide to deploying a Zookeeper‑Kafka‑Filebeat‑ClickHouse pipeline, including common pitfalls and their solutions, showing how the combo reduces server costs by up to half.

ClickHouseCost ReductionElasticsearch
0 likes · 15 min read
Why a CK+Kafka+Filebeat Stack Beats ELK for Log Analytics
Java Tech Enthusiast
Java Tech Enthusiast
Aug 8, 2026 · Backend Development

Why Does Kafka Re‑Consume Tens of Thousands of Messages After a Successful Offset Commit?

The article explains why Kafka consumers can reprocess tens of thousands of messages after a restart despite successful offset commits, detailing the roles of automatic commit timing, rebalance triggers, GC pauses, and the limitations of manual commits, and offers practical configuration and idempotent processing solutions.

ConsumerJavaKafka
0 likes · 8 min read
Why Does Kafka Re‑Consume Tens of Thousands of Messages After a Successful Offset Commit?
Cloud Architecture
Cloud Architecture
Aug 7, 2026 · Backend Development

Building a Trillion‑Message Queue: Kafka‑Level Architecture and Implementation

This article explains why and how to build a Kafka‑grade message‑queue kernel from scratch, detailing functional and non‑functional goals, core design principles, storage layout, replication, consumer‑group coordination, performance optimizations, deployment on Kubernetes, and a step‑by‑step roadmap to production‑grade reliability.

JavaKafkaMessage Queue
0 likes · 40 min read
Building a Trillion‑Message Queue: Kafka‑Level Architecture and Implementation
Cloud Architecture
Cloud Architecture
Aug 6, 2026 · Big Data

Exporting 10 Billion Elasticsearch Records: From Simple Script to Enterprise Offline Platform

The article analyses why exporting billions of Elasticsearch documents requires a full‑stack platform rather than a one‑off script, detailing the pitfalls of naive pagination, the benefits of PIT + search_after + slicing, and a complete architecture with Kafka, Redis, MySQL, Kubernetes and observability for reliable, scalable offline data export.

ElasticsearchJavaKafka
0 likes · 40 min read
Exporting 10 Billion Elasticsearch Records: From Simple Script to Enterprise Offline Platform
Rain's Path to Java Mastery
Rain's Path to Java Mastery
Aug 5, 2026 · Backend Development

MQ Ordered Consumption: Partial Order Patterns, Code Examples & Interview Guide

This article explains how to guarantee message ordering in message queues by comparing RocketMQ, Kafka, and RabbitMQ implementations, detailing partial ordering via business key hashing, providing production-ready code for producers and consumers, and covering business-level safeguards like state machines and idempotency to handle queue rebalancing and head-of-line blocking.

Head-of-line BlockingIdempotencyKafka
0 likes · 25 min read
MQ Ordered Consumption: Partial Order Patterns, Code Examples & Interview Guide
Code Farming
Code Farming
Aug 4, 2026 · Backend Development

Why 50,000 Simultaneous Registrations Crashed the System—and How Isolation Prevents It

When an education company faced 50,000 concurrent re‑registration requests, its monolithic internal‑external architecture collapsed, but by physically separating networks, enforcing gateway rate‑limiting and circuit‑breaking, pre‑loading data into Redis, and using one‑way Kafka streams, the system remained stable.

Backend ArchitectureCircuit BreakingKafka
0 likes · 6 min read
Why 50,000 Simultaneous Registrations Crashed the System—and How Isolation Prevents It
YiSu Grain
YiSu Grain
Jul 28, 2026 · Big Data

Day 39: Big Data Architecture – Distributed Storage, Batch vs Stream Processing, and Compute‑Storage Separation

This lesson explains why a single database cannot scale for massive e‑commerce data, introduces the three core pillars of distributed storage—sharding, replication, and horizontal scaling—covers batch and stream processing differences with Spark, Hive, Flink and Storm, compares compute‑storage integration versus separation, and shows how Kafka, HDFS, Spark and Flink fit together in a real‑time data platform.

Batch ProcessingFlinkKafka
0 likes · 25 min read
Day 39: Big Data Architecture – Distributed Storage, Batch vs Stream Processing, and Compute‑Storage Separation
Smart Sea Tide
Smart Sea Tide
Jul 27, 2026 · Big Data

Comprehensive Overview of Big Data Open‑Source Frameworks

This article provides a detailed, structured survey of the most widely used open‑source big‑data technologies—including Hadoop ecosystems, storage systems, processing engines, query tools, data ingestion, exchange, messaging, scheduling, governance, visualization, mining, and cloud platforms—highlighting each project's origins, core features, typical use cases, and notable strengths or limitations to aid technology selection and system design.

FlinkHBaseHadoop
0 likes · 59 min read
Comprehensive Overview of Big Data Open‑Source Frameworks
Ray's Galactic Tech
Ray's Galactic Tech
Jul 17, 2026 · Backend Development

Three Critical Guarantees for Message Queues: No Loss, No Duplicates, No Disorder – Deep Dive into Production‑Grade Solutions

This article dissects why modern systems must enforce three reliability guarantees—no message loss, no duplicate processing, and no out‑of‑order delivery—by examining real‑world order flows, outbox patterns, idempotent keys, partitioning strategies, consumer acknowledgments, and operational safeguards such as replay, dead‑letter handling, and monitoring.

Distributed SystemsIdempotencyKafka
0 likes · 28 min read
Three Critical Guarantees for Message Queues: No Loss, No Duplicates, No Disorder – Deep Dive into Production‑Grade Solutions
Ray's Galactic Tech
Ray's Galactic Tech
Jul 16, 2026 · Backend Development

Building a Scalable Smart Tag System for 100k QPS with AI‑Generated Code

The article explains that while many teams focus on model accuracy, the real challenges of an AI‑powered tagging system are write spikes, timeouts, duplicate tagging, cost overruns and state inconsistency, and it proposes a five‑layer architecture, async fallback, governance and validation practices to reliably achieve a 100 k QPS target.

AI taggingKafkaLLM integration
0 likes · 30 min read
Building a Scalable Smart Tag System for 100k QPS with AI‑Generated Code
Ray's Galactic Tech
Ray's Galactic Tech
Jul 16, 2026 · Artificial Intelligence

K8s, Kafka, Nacos Agent Platform to Prevent Token Bankruptcy and Skill Avalanches

The article details how a production‑grade Agent platform built on Kubernetes, Kafka, and Nacos addresses token budget overruns, uncontrolled skill execution, and RAG hallucinations by introducing a four‑layer runtime architecture, token pre‑allocation, explicit state management, dynamic governance policies, and robust skill specifications.

KafkaLLM agentsNacos
0 likes · 32 min read
K8s, Kafka, Nacos Agent Platform to Prevent Token Bankruptcy and Skill Avalanches
Xiaolin Talks Programming
Xiaolin Talks Programming
Jul 16, 2026 · Backend Development

CDC + Outbox Pattern: Production-Grade Cross-Database Sync with Spring Boot & Kafka

This article details a production-ready cross-database synchronization architecture using CDC and Outbox pattern with Spring Boot, Debezium, and Kafka, covering schema design, connector configuration, transaction boundaries, ordering guarantees, idempotent consumption, dead-letter queues, and monitoring strategies.

CDCCross-Database SyncDebezium
0 likes · 20 min read
CDC + Outbox Pattern: Production-Grade Cross-Database Sync with Spring Boot & Kafka
Ray's Galactic Tech
Ray's Galactic Tech
Jul 14, 2026 · Cloud Native

Spring Boot + Netty MQTT Gateway for Million Connections & Millisecond Push

To support millions of persistent MQTT connections with sub‑millisecond latency, the article walks through a Spring Boot + Netty cloud‑native gateway design that separates connection, event, state and governance planes, details async authentication, back‑pressure handling, command state machines, and loss‑less Kubernetes roll‑outs.

Cloud NativeKafkaMQTT
0 likes · 38 min read
Spring Boot + Netty MQTT Gateway for Million Connections & Millisecond Push
dbaplus Community
dbaplus Community
Jul 13, 2026 · Interview Experience

How to Process a 10 GB CSV File with Only 512 MB RAM?

The article walks through an interview scenario where a candidate must handle a 10 GB CSV using just 512 MB of memory, critiques naive answers, and presents a step‑by‑step streaming, chunked, and checkpoint‑based solution with optional Kafka/Flink considerations.

CSVFlinkJava
0 likes · 8 min read
How to Process a 10 GB CSV File with Only 512 MB RAM?
Xiaolin Talks Programming
Xiaolin Talks Programming
Jul 13, 2026 · Backend Development

CQRS & Event Sourcing in Spring Boot: Read/Write Separation, Snapshots & Compensation

This guide details implementing CQRS and Event Sourcing in Spring Boot, covering command/query separation, event bus choices (Outbox pattern, Kafka), projection idempotency, state snapshots for performance, Saga-based compensation for distributed transactions, and practical tuning tips for production workloads.

CQRSEvent BusIdempotency
0 likes · 14 min read
CQRS & Event Sourcing in Spring Boot: Read/Write Separation, Snapshots & Compensation
Niu Liu
Niu Liu
Jul 13, 2026 · Artificial Intelligence

Building a Real‑Time Recommendation Engine with Flink: A Complete Example Project

The article walks through constructing a full‑stack real‑time recommendation system—from user‑behavior collection via Kafka, through Flink streaming jobs for hot‑list, user and item profiling, to storage in Redis, HBase and Elasticsearch, and finally a React/Ant Design console that visualizes the pipeline and enables debugging.

ElasticsearchFlinkHBase
0 likes · 12 min read
Building a Real‑Time Recommendation Engine with Flink: A Complete Example Project
TechVision Expert Circle
TechVision Expert Circle
Jul 12, 2026 · Backend Development

Designing a Millisecond‑Level Real‑Time Data Processing System

The article outlines practical engineering steps to build a sub‑200 ms end‑to‑end real‑time data pipeline, covering latency budgeting, data ingestion with gRPC/Kafka, stream engine selection (Flink 2.x, RisingWave), state backend tuning, output sink choices, and monitoring with backpressure and OpenTelemetry.

Apache FlinkKafkaOpenTelemetry
0 likes · 13 min read
Designing a Millisecond‑Level Real‑Time Data Processing System
Cloud Architecture
Cloud Architecture
Jul 9, 2026 · Backend Development

How Spring Boot, Kafka, Redis, and MongoDB Power Real‑Time GPS Tracking for Millions of Vehicles

This article walks through a production‑grade architecture that uses Spring Boot, Kafka, Redis, and MongoDB to ingest, buffer, order, and store high‑frequency vehicle GPS data from hundreds of thousands of devices while guaranteeing low latency, scalability, fault‑tolerance, and accurate replay capabilities.

KafkaMongoDBRedis
0 likes · 38 min read
How Spring Boot, Kafka, Redis, and MongoDB Power Real‑Time GPS Tracking for Millions of Vehicles
YiSu Grain
YiSu Grain
Jul 7, 2026 · Fundamentals

Why You Don't Need to Send an SMS Immediately After an Order – Understanding Kafka and Message Queues

The article explains how using a message queue such as Kafka lets an order service return quickly by handling non‑critical tasks like SMS, points, and logging asynchronously, thereby improving latency, decoupling services, and smoothing traffic spikes while also covering Kafka's core concepts and trade‑offs.

Asynchronous ProcessingDecouplingKafka
0 likes · 11 min read
Why You Don't Need to Send an SMS Immediately After an Order – Understanding Kafka and Message Queues
Su San Talks Tech
Su San Talks Tech
Jul 7, 2026 · Backend Development

How to Choose Among Four Popular Message Queues?

This article compares Kafka, RocketMQ, RabbitMQ, and ActiveMQ, explaining their architectures, core concepts, strengths and weaknesses, and provides practical guidance on selecting the most suitable queue based on throughput, reliability, scalability, and use‑case requirements.

Backend DevelopmentKafkaMessage Queue
0 likes · 20 min read
How to Choose Among Four Popular Message Queues?
Architect Chen
Architect Chen
Jul 6, 2026 · Big Data

Comprehensive Guide to Kafka Core Commands (2026 Edition)

This article walks through the essential Kafka command‑line tools, covering how to check the Kafka version, create, list, describe, alter, and delete topics, manage partitions, produce and consume messages, inspect consumer groups, reset offsets, query broker metadata, view log directories, and run built‑in performance tests for both producers and consumers.

Consumer GroupKafkaPerformance Testing
0 likes · 6 min read
Comprehensive Guide to Kafka Core Commands (2026 Edition)
Lobster Programming
Lobster Programming
Jul 6, 2026 · Backend Development

Why Kafka Achieves Such High Throughput

Kafka’s exceptionally high throughput stems from a combination of design choices—including sequential disk writes, zero‑copy data transfer, batch processing, partition‑level parallelism, OS page‑cache utilization, and producer‑side compression—that together minimize I/O overhead, CPU usage, and latency.

KafkaPartitioningbatching
0 likes · 6 min read
Why Kafka Achieves Such High Throughput
Java Architect Handbook
Java Architect Handbook
Jul 4, 2026 · Backend Development

How to Build a Clean SpringBoot Scaffold Quickly

The article walks through creating a SpringBoot project from scratch, covering IDE setup pain points, project initialization, version compatibility between SpringBoot, Spring Cloud and Kafka, Maven dependency management, core scaffold classes such as global exception handling, logging aspect, CORS and Swagger configuration, and recommends useful tools like embedded Redis, MariaDB, Hutool, MyBatis‑Plus, MapStruct and Redisson.

KafkaMapStructMaven
0 likes · 11 min read
How to Build a Clean SpringBoot Scaffold Quickly
Random Bulletin
Random Bulletin
Jul 1, 2026 · Operations

Evolving Message Expiration for 10 Million QPS: From No TTL to Smart Policies

When a high‑traffic system processes billions of messages per day, stale “zombie” messages can corrupt business state; this article walks through the evolution from never‑expiring queues to uniform TTL, then per‑topic and per‑message TTL, and finally to smart, context‑aware expiration, detailing the engineering trade‑offs, implementation patterns, and operational checklist needed for reliable 10 M‑QPS message pipelines.

Distributed SystemsKafkaMessage Queue
0 likes · 19 min read
Evolving Message Expiration for 10 Million QPS: From No TTL to Smart Policies
Java Baker
Java Baker
Jun 29, 2026 · Backend Development

How to Diagnose Uneven CPU Usage in Java Services Using Kafka

This article walks through the symptoms, root cause analysis, and step‑by‑step solutions for uneven CPU usage across Java service instances, highlighting how mismatched Kafka partition counts and thread or GC issues can lead to load imbalance and how to resolve them.

CPUJavaKafka
0 likes · 8 min read
How to Diagnose Uneven CPU Usage in Java Services Using Kafka
Cloud Architecture
Cloud Architecture
Jun 28, 2026 · Backend Development

Building a Production‑Grade Kafka Reverse Proxy with Nginx + Lua for Scalable, Auditable Message Access

The article explains why a simple TCP proxy is insufficient for Kafka, and presents a production‑ready reverse‑proxy architecture built on Nginx stream and Lua that adds metadata rewriting, multi‑tenant quota, TLS termination, dynamic routing, observability, and graceful degradation to achieve scalable, auditable message ingestion.

KafkaLuaMessage Governance
0 likes · 33 min read
Building a Production‑Grade Kafka Reverse Proxy with Nginx + Lua for Scalable, Auditable Message Access
Random Bulletin
Random Bulletin
Jun 28, 2026 · Operations

Backlog Monitoring at 10 Million QPS: From Manual Checks to SLO‑Driven Automation

At 10 million QPS, message backlog can silently grow for hours, turning minutes of lag into hundreds of millions of undelivered messages; this article walks through a six‑stage evolution—from manual command‑line checks to SLO‑driven predictive monitoring—detailing metrics, pitfalls, and tool choices for a robust, multi‑dimensional alert system.

KafkaSLOalerting
0 likes · 21 min read
Backlog Monitoring at 10 Million QPS: From Manual Checks to SLO‑Driven Automation
Cloud Architecture
Cloud Architecture
Jun 27, 2026 · Backend Development

Enterprise Messaging System Deep Dive: Core Engine to High‑Concurrency

This guide explains how enterprise messaging systems reliably propagate state changes across distributed services, covering core concepts, outbox patterns, topic/tag modeling, consumer idempotency, dead‑letter handling, high‑concurrency engineering, monitoring, and Kubernetes deployment to build a production‑grade, observable platform.

KafkaOutboxevent-driven
0 likes · 44 min read
Enterprise Messaging System Deep Dive: Core Engine to High‑Concurrency
DeepNoMind
DeepNoMind
Jun 27, 2026 · Backend Development

Designing a Production‑Grade Distributed Logging and Metrics Platform

This article presents an end‑to‑end design of a production‑grade observability platform that ingests millions of real‑time logs, metrics, and events, detailing functional and non‑functional requirements, capacity planning, component choices such as Kafka, Flink, Elasticsearch, object‑storage data lakes, and the trade‑offs involved.

ElasticsearchFlinkKafka
0 likes · 21 min read
Designing a Production‑Grade Distributed Logging and Metrics Platform
Alibaba Cloud Native
Alibaba Cloud Native
Jun 26, 2026 · Cloud Native

One-Click Real-Time Stream Ingestion: Alibaba Cloud Kafka’s Native Data Lake Integration

Alibaba Cloud Message Queue for Kafka introduces a native message‑to‑lake capability that integrates Apache Iceberg with OSS Table Bucket, eliminating Spark/Flink/Kafka Connect, providing exactly‑once semantics, automatic schema management, dual write modes, smart partitioning, and up to ten‑fold performance gains across diverse real‑time analytics scenarios.

Apache IcebergCloud NativeKafka
0 likes · 12 min read
One-Click Real-Time Stream Ingestion: Alibaba Cloud Kafka’s Native Data Lake Integration
Random Bulletin
Random Bulletin
Jun 24, 2026 · Operations

Rack Awareness: From Zero to High‑QPS – Boost Availability, Cut Cross‑AZ Traffic

A real‑world rack‑power outage showed that three‑replica Kafka clusters can lose all replicas when brokers share a failure domain, prompting a deep dive into rack awareness—how fault‑domain tags are injected, replica‑placement and leader‑distribution algorithms, consumer‑proximity reads, bandwidth costs, failure scenarios, and the stepwise evolution from hundred‑thousand to ten‑million QPS deployments.

Distributed SystemsKafkaPulsar
0 likes · 26 min read
Rack Awareness: From Zero to High‑QPS – Boost Availability, Cut Cross‑AZ Traffic
Random Bulletin
Random Bulletin
Jun 23, 2026 · Backend Development

Choosing the Right Replication Factor for 10M QPS Message Queues: From Dual to Multi‑Replica

The article walks through why a single replica is insufficient for million‑scale message queues, explains the latency and data‑loss trade‑offs of dual‑replica sync and async modes, shows how three‑replica majority voting becomes the sweet spot, and then details the engineering considerations for scaling to five or more replicas across availability zones and regions.

AZ awarenessDistributed SystemsHigh Availability
0 likes · 18 min read
Choosing the Right Replication Factor for 10M QPS Message Queues: From Dual to Multi‑Replica
Programmer XiaoFu
Programmer XiaoFu
Jun 23, 2026 · Backend Development

Why Kafka Still Delivers Out‑of‑Order Messages Even When Using the Same Key

Even though Kafka guarantees that messages with the same key land in the same partition, the article explains how producer retries, multithreaded consumer processing, and partition expansion can break ordering, and provides concrete techniques such as idempotent producers and single‑threaded consumption to preserve order.

Consumer ConcurrencyKafkaPartition Expansion
0 likes · 10 min read
Why Kafka Still Delivers Out‑of‑Order Messages Even When Using the Same Key
ZhiKe AI
ZhiKe AI
Jun 22, 2026 · Fundamentals

Message Queues: Power When Correct, Disaster When Wrong – 3 Scenarios & Tips

The article explains how message queues can dramatically improve response time, decouple services, and smooth traffic spikes, outlines seven advantages and eight drawbacks, and provides concrete guidelines on when to adopt them, how to prevent loss, duplication, and ordering issues, and how to ensure end‑to‑end reliability.

Asynchronous ProcessingIdempotencyKafka
0 likes · 15 min read
Message Queues: Power When Correct, Disaster When Wrong – 3 Scenarios & Tips
Random Bulletin
Random Bulletin
Jun 20, 2026 · Backend Development

Message Ordering at 10M+ QPS: From Global to Partition to Business-Key

The article explains that message ordering is a spectrum—from strict global ordering to partition-level and finally business-key ordering—and examines the trade‑offs, pitfalls, and engineering techniques needed to maintain order while scaling to tens of millions of QPS.

KafkaPartitioningbusiness key
0 likes · 24 min read
Message Ordering at 10M+ QPS: From Global to Partition to Business-Key
Niu Liu
Niu Liu
Jun 19, 2026 · Big Data

Building a Real‑Time Risk Control Engine with Flink 2.2.1, CEP, and Aviator

This article details a real‑time risk control system for e‑commerce and finance built on Apache Flink 2.2.1 and CEP, featuring a dynamic Aviator rule engine, three Kafka event streams, multi‑channel output to Redis, MySQL and Kafka, a Spring Boot‑React management UI, and step‑by‑step deployment instructions.

AviatorCEPFlink
0 likes · 11 min read
Building a Real‑Time Risk Control Engine with Flink 2.2.1, CEP, and Aviator
Architect Chen
Architect Chen
Jun 19, 2026 · Operations

Comprehensive Guide to Kafka Commands (2026 Edition)

This article provides a step‑by‑step reference for Kafka command‑line tools, covering version checks, topic creation, listing, describing, deletion, partition alteration, consumer‑group inspection, offset resets, producer/consumer testing, broker API version queries, cluster metadata, leader election, and log‑directory monitoring, with concrete examples and expected outputs.

KafkaTopic Managementbroker monitoring
0 likes · 6 min read
Comprehensive Guide to Kafka Commands (2026 Edition)
Alibaba Cloud Native
Alibaba Cloud Native
Jun 19, 2026 · Big Data

Why Real-Time Data Lake Ingestion Is Dropping ETL in the AI Era: Architecture Simplification from Kafka to Iceberg

In the AI‑driven era, enterprises need a data foundation that supports both real‑time consumption and long‑term historical analysis, and the emerging "zero‑ETL" trend moves generic ingestion capabilities from external Flink/Spark jobs into a streamlined Kafka‑to‑Iceberg pipeline, reducing complexity while preserving low latency, consistency, schema evolution, CDC semantics and open‑ecosystem compatibility.

IcebergKafkaTable Bucket
0 likes · 25 min read
Why Real-Time Data Lake Ingestion Is Dropping ETL in the AI Era: Architecture Simplification from Kafka to Iceberg