Tagged articles

Kafka

1415 articles · Page 3 of 15
Efficient Ops
Efficient Ops
Oct 12, 2025 · Information Security

What Exposed the AI Companion Apps? A Deep Dive into the Massive Data Leak

A recent breach of the AI companion apps Chattee and GiMe Chat exposed over 40 million private conversations, hundreds of thousands of media files, and transaction records due to publicly accessible Kafka brokers lacking any authentication, putting user privacy and security at serious risk.

AI securityKafkadata breach
0 likes · 3 min read
What Exposed the AI Companion Apps? A Deep Dive into the Massive Data Leak
ITPUB
ITPUB
Oct 5, 2025 · Backend Development

How to Clear a 10‑Million‑Message Queue in 5 Hours: A Five‑Step Rescue Plan

When a flash‑sale causes a 10 million‑message backlog and consumers only process 200 messages per second, this guide shows a five‑step, 5‑hour strategy—horizontal scaling, message downgrade, flow control, temporary dump, and parallel blasting—to restore throughput and prevent system collapse.

KafkaPerformance Optimizationdistributed systems
0 likes · 6 min read
How to Clear a 10‑Million‑Message Queue in 5 Hours: A Five‑Step Rescue Plan
Lakehouse Research Base
Lakehouse Research Base
Oct 3, 2025 · Operations

100 Advanced Linux Interview Questions for Big Data Operations

This comprehensive guide presents 100 advanced Linux interview questions tailored for big data operations, covering command-line mastery, system performance tuning, cluster configuration for Hadoop, Kafka, Spark, and Flink, complex troubleshooting scenarios, automation with Ansible and Prometheus, security hardening, and containerized deployments on Kubernetes.

HadoopKafkaLinux
0 likes · 45 min read
100 Advanced Linux Interview Questions for Big Data Operations
ITPUB
ITPUB
Oct 3, 2025 · Big Data

How Qunar Travel Cut 2000 CPU Cores by Optimizing Kafka Production

This case study details how Qunar Travel's engineering team analyzed Kafka production bottlenecks during peak traffic, added targeted monitoring, tuned thread and batch parameters, and validated the changes through gray‑scale tests, ultimately saving about 2000 CPU cores across three clusters while reducing request volume and improving network and disk utilization.

CPU SavingsKafkaKubernetes
0 likes · 14 min read
How Qunar Travel Cut 2000 CPU Cores by Optimizing Kafka Production
Cognitive Technology Team
Cognitive Technology Team
Oct 3, 2025 · Backend Development

Mastering Kafka Consumer Rebalance: Strategies to Boost Throughput and Stability

This article deeply explores Kafka consumer group rebalance mechanisms, identifies performance pitfalls of frequent rebalances, and provides a comprehensive set of configuration tweaks, assignment strategies, batch processing techniques, and monitoring practices to achieve a more stable and high‑throughput Kafka consumer system.

BackendConsumerKafka
0 likes · 16 min read
Mastering Kafka Consumer Rebalance: Strategies to Boost Throughput and Stability
Lakehouse Research Base
Lakehouse Research Base
Oct 2, 2025 · Operations

100 Linux Interview Questions for Big Data Operations

This comprehensive guide presents 100 Linux interview questions tailored for big data operations, covering basic commands, permissions, process management, networking, log processing, shell scripting, and cluster-specific operations with detailed answers and practical explanations.

HadoopKafkaLinux
0 likes · 41 min read
100 Linux Interview Questions for Big Data Operations
IT Services Circle
IT Services Circle
Oct 1, 2025 · Backend Development

Designing a Fast, Reliable, Cost‑Effective Like System for High‑Traffic Apps

This article breaks down the essential requirements and architecture of a high‑performance like system, covering fast response, data consistency, scalability under traffic spikes, and cost‑efficient resource use, while detailing the technical stack, caching strategies, async persistence, and practical optimizations.

KafkaPerformancebackend design
0 likes · 17 min read
Designing a Fast, Reliable, Cost‑Effective Like System for High‑Traffic Apps
Architecture Digest
Architecture Digest
Sep 23, 2025 · Backend Development

How to Ensure Zero Message Loss in Kafka: Proven Strategies for High‑Reliability Systems

This article explains Kafka's storage architecture, identifies three major message‑loss scenarios across production, storage, and consumption, and provides practical end‑to‑end configurations, detection methods, and business‑level patterns to achieve near‑zero message loss in high‑concurrency distributed systems.

KafkaMessage reliabilitydata consistency
0 likes · 13 min read
How to Ensure Zero Message Loss in Kafka: Proven Strategies for High‑Reliability Systems
Su San Talks Tech
Su San Talks Tech
Sep 23, 2025 · Backend Development

How to Guarantee 100% Message Delivery with Kafka: Interview‑Ready Strategies

This article dissects Kafka’s storage architecture, identifies loss points in production, storage, and consumption phases, and presents interview‑ready strategies—including acks settings, flush tuning, consumer batch commits, detection via sequence numbers, and transactional messaging—to guarantee virtually 100 % message durability.

Consumer CommitKafkaTransactional Messaging
0 likes · 20 min read
How to Guarantee 100% Message Delivery with Kafka: Interview‑Ready Strategies
Ray's Galactic Tech
Ray's Galactic Tech
Sep 21, 2025 · Cloud Native

How to Deploy a Scalable Kafka Cluster on Bare Metal and Kubernetes

This guide walks you through installing a high‑availability Kafka cluster, covering both bare‑metal/virtual‑machine setups and Kubernetes‑based deployments with Helm, including environment preparation, ZooKeeper configuration, broker settings, Helm chart customization, KRaft mode, and production‑grade tuning recommendations.

HelmKRaftKafka
0 likes · 7 min read
How to Deploy a Scalable Kafka Cluster on Bare Metal and Kubernetes
Su San Talks Tech
Su San Talks Tech
Sep 16, 2025 · Backend Development

Mastering Message Order in Distributed Queues: From Basics to Advanced Strategies

This article explores the fundamentals of message ordering in distributed message queues, explains why ordering is determined by broker arrival, compares global and partial ordering, and presents practical solutions—from single-partition designs to multi-partition hashing, handling data skew, and safe expansion—plus interview tips.

KafkaPartitioningconsistent hashing
0 likes · 24 min read
Mastering Message Order in Distributed Queues: From Basics to Advanced Strategies
Mike Chen's Internet Architecture
Mike Chen's Internet Architecture
Sep 12, 2025 · Backend Development

How to Prevent Kafka Message Loss: Producer, Broker, and Consumer Strategies

This guide explains comprehensive Kafka message loss prevention techniques, covering producer configurations like acks=all, retries, idempotence, broker settings such as replication factor and min.insync.replicas, consumer handling with manual offset commits, and full‑stack compensation and retry mechanisms with code examples.

Broker SettingsConsumer offsetKafka
0 likes · 3 min read
How to Prevent Kafka Message Loss: Producer, Broker, and Consumer Strategies
mikechen
mikechen
Sep 11, 2025 · Backend Development

How Does Kafka Guarantee Message Order? Key Mechanisms Explained

This article explains why message ordering is critical in scenarios such as financial transactions, e‑commerce order updates, audit logs, and IoT devices, then details Kafka’s core components—topics, partitions, offsets, producers, brokers, and consumer groups—and describes how single‑partition consumption and consistent key hashing ensure ordered processing while balancing throughput.

BackendConsumer GroupKafka
0 likes · 5 min read
How Does Kafka Guarantee Message Order? Key Mechanisms Explained
Architect Chen
Architect Chen
Sep 10, 2025 · Big Data

How Kafka Achieves Million‑Message Throughput: Sequential Writes, Page Cache, Batching & Zero‑Copy

The article explains how Kafka attains high‑throughput performance by using sequential disk writes, leveraging the OS page cache, employing producer and consumer batching with configurable parameters, and utilizing zero‑copy sendfile to minimize CPU and memory overhead, enabling stable million‑message per second rates.

KafkaSequential Writebatching
0 likes · 5 min read
How Kafka Achieves Million‑Message Throughput: Sequential Writes, Page Cache, Batching & Zero‑Copy
IT Architects Alliance
IT Architects Alliance
Sep 8, 2025 · Cloud Native

Choosing the Right Message Queue: Kafka vs RabbitMQ vs Pulsar – A Practical Guide

This article examines the core evaluation criteria for enterprise message queues and provides a detailed comparison of Kafka, RabbitMQ, and Pulsar—including performance, reliability, scalability, operational complexity, ecosystem maturity, and business fit—offering actionable guidance for selecting the optimal solution in cloud‑native environments.

KafkaPulsarRabbitMQ
0 likes · 9 min read
Choosing the Right Message Queue: Kafka vs RabbitMQ vs Pulsar – A Practical Guide
Ops Community
Ops Community
Sep 8, 2025 · Operations

Mastering Distributed Log Architecture: From Flume to ELK and Beyond

This comprehensive guide walks you through the challenges of large‑scale log collection, real‑time processing, storage optimization, and visualization, detailing practical configurations for Flume, Logstash, Elasticsearch, Kibana, Filebeat, Kafka, Kubernetes, and future AIOps integrations to build a reliable, cost‑effective distributed logging system.

ELKFlumeKafka
0 likes · 24 min read
Mastering Distributed Log Architecture: From Flume to ELK and Beyond
Smart Sea Tide
Smart Sea Tide
Sep 8, 2025 · Big Data

How to Migrate Data from HBase to Kafka: A Reverse Data Flow Guide

This article explains how to move data from an HBase cluster back into Kafka by extracting RowKeys with a MapReduce job, handling massive volumes and lack of partitioning, and reliably writing the records to Kafka while tracking successes and failures.

FirstKeyOnlyFilterHBaseHDFS
0 likes · 9 min read
How to Migrate Data from HBase to Kafka: A Reverse Data Flow Guide
IT Services Circle
IT Services Circle
Sep 6, 2025 · Backend Development

10 Real‑World Scenarios Where Message Queues Transform Your System

This article explores ten practical use‑cases for message queues—covering system decoupling, asynchronous processing, traffic shaping, data synchronization, log collection, broadcast updates, ordered and delayed messages, retry mechanisms, and transactional messaging—illustrated with Java code examples and architectural diagrams.

Backend DevelopmentJavaKafka
0 likes · 17 min read
10 Real‑World Scenarios Where Message Queues Transform Your System
Mike Chen's Internet Architecture
Mike Chen's Internet Architecture
Sep 4, 2025 · Backend Development

How to Diagnose and Fix Kafka Message Backlog Issues

This article explains what Kafka message backlog is, why it harms performance and stability, and provides practical producer‑side throttling and consumer‑side scaling techniques, illustrated with diagrams, to help engineers eliminate backlog and keep their Kafka clusters healthy.

Kafkaconsumer scalingmessage backlog
0 likes · 4 min read
How to Diagnose and Fix Kafka Message Backlog Issues
Tech Freedom Circle
Tech Freedom Circle
Sep 4, 2025 · Backend Development

How to Solve ES Latency in MySQL‑Canal Sync and Indexing Scenarios?

The article dissects the interview question about ES latency in a MySQL‑Canal‑to‑Elasticsearch pipeline, explains the root causes across four system layers, and presents a comprehensive four‑layer optimization, end‑to‑end observability, routing‑based degradation, and a Java‑based LatencyProbe component to measure and control delay.

CanalElasticsearchKafka
0 likes · 17 min read
How to Solve ES Latency in MySQL‑Canal Sync and Indexing Scenarios?
ITPUB
ITPUB
Sep 3, 2025 · Backend Development

How We Boosted Kafka Throughput by 35% with Filebeat Tuning and Compression Tricks

This case study details how a high‑traffic Kafka logging cluster was optimized by analyzing low compression ratios, tuning Filebeat parameters, adjusting memory queues and round‑robin settings, and validating the changes through gray‑scale tests, resulting in up to 35% higher throughput and significant resource savings.

CompressionFilebeatKafka
0 likes · 10 min read
How We Boosted Kafka Throughput by 35% with Filebeat Tuning and Compression Tricks
php Courses
php Courses
Aug 29, 2025 · Operations

How to Build a Real‑Time PHP Log Event Pipeline for Instant Insights

Learn how to transform PHP logs into real‑time, structured events by implementing a log event pipeline that includes JSON logging, lightweight collectors like Filebeat, streaming platforms such as Kafka or Flink, enrichment, and visualization with Grafana, enabling instant monitoring, alerting, and data‑driven decisions.

FlinkGrafanaKafka
0 likes · 7 min read
How to Build a Real‑Time PHP Log Event Pipeline for Instant Insights
Ubiquitous Tech
Ubiquitous Tech
Aug 26, 2025 · Backend Development

Java Interview Simulation: Solving Sohu’s High‑Throughput Push System

This article walks through a five‑round Java interview simulation where the candidate designs, optimizes, and secures a massive push‑notification system for Sohu, covering Spring Boot high‑concurrency architecture, sharding and cache‑penetration mitigation, Kafka reliability, Kubernetes deployment with gray releases, multi‑region high‑availability, and AI‑driven smart‑push strategies, complete with concrete code snippets and real‑world performance numbers.

KafkaKubernetesRedis
0 likes · 37 min read
Java Interview Simulation: Solving Sohu’s High‑Throughput Push System
Ubiquitous Tech
Ubiquitous Tech
Aug 24, 2025 · Backend Development

Java Interview Deep Dive: Solving Real‑World Invoice System Challenges

The article walks through a simulated five‑round Java interview where the candidate designs a high‑concurrency invoice‑issuing service, covering gateway architecture, async Kafka processing, cache strategies, JVM thread‑pool and memory tuning, database‑cache coordination, microservice messaging, system design, performance optimization and AI integration, all illustrated with concrete numbers and code snippets.

JVMJavaKafka
0 likes · 35 min read
Java Interview Deep Dive: Solving Real‑World Invoice System Challenges
Su San Talks Tech
Su San Talks Tech
Aug 23, 2025 · Backend Development

How to Build a Robust Asynchronous Processing SDK with Spring and Kafka

This article explains how to design and implement a generic asynchronous processing SDK for Java backend services, covering its purpose, advantages, underlying principles, component choices, database schema, configuration, usage patterns, and best‑practice notes, with code examples and diagrams.

ConfigurationJava backendKafka
0 likes · 9 min read
How to Build a Robust Asynchronous Processing SDK with Spring and Kafka
Alibaba Cloud Big Data AI Platform
Alibaba Cloud Big Data AI Platform
Aug 21, 2025 · Big Data

How Hypergryph Built a High‑Performance Real‑Time Analytics Platform with StarRocks

This case study details how Hypergryph leveraged Alibaba Cloud EMR Serverless StarRocks, Flink, and Kafka to replace a ClickHouse data warehouse with a high‑performance, elastic, and easy‑to‑operate real‑time analytics platform that dramatically improved query speed, stability, operational efficiency, and cost for their gaming business.

Cloud ComputingFlinkKafka
0 likes · 8 min read
How Hypergryph Built a High‑Performance Real‑Time Analytics Platform with StarRocks
Mike Chen's Internet Architecture
Mike Chen's Internet Architecture
Aug 20, 2025 · Backend Development

How Kafka Achieves High‑Performance Storage: Log Segments, Indexes, and Retention

This article explains Kafka's storage architecture, including its partition‑based log files, sequential append writes, log segment management, index files, and configurable time‑ and size‑based retention policies that together enable ultra‑high write throughput while controlling disk usage.

KafkaRetention PolicySequential Write
0 likes · 4 min read
How Kafka Achieves High‑Performance Storage: Log Segments, Indexes, and Retention
MaGe Linux Operations
MaGe Linux Operations
Aug 19, 2025 · Big Data

Master Kafka High Availability: Replica Sync & Disaster Recovery Strategies

This article provides a comprehensive guide to building enterprise‑grade, highly available Kafka clusters, covering architecture design, hardware planning, production‑level broker configurations, ISR management, monitoring, fault‑tolerance procedures, rolling upgrades, capacity planning, and automation scripts for seamless operations.

KafkaOperationsdisaster-recovery
0 likes · 16 min read
Master Kafka High Availability: Replica Sync & Disaster Recovery Strategies
StarRocks
StarRocks
Aug 19, 2025 · Big Data

How Joydata Scaled to 150 Billion Daily Events with StarRocks: A Data Architecture Journey

Facing daily data growth from millions to 150 billion records, Joydata‑U transformed its analytics platform through three architectural stages—Hadoop, Hadoop + Trino, and finally StarRocks—introducing resource isolation, Flat JSON acceleration, and Bitmap indexing to cut query latency by up to seven times and achieve sub‑2‑minute data freshness across BI, ad‑tech, game analytics, and CRM workloads.

Bitmap IndexFlat JSONFlink
0 likes · 12 min read
How Joydata Scaled to 150 Billion Daily Events with StarRocks: A Data Architecture Journey
Tencent Cloud Middleware
Tencent Cloud Middleware
Aug 1, 2025 · Backend Development

Mastering TDMQ CKafka Production: Configurations, Partitioning, and Reliable Messaging

This guide explains how to efficiently produce messages with TDMQ CKafka, covering topic creation, partition sizing, retry policies, asynchronous sending, key/value handling, batch tuning, sticky partitioning, idempotence, ACK settings, code examples, and strategies to avoid data skew and ensure high‑throughput, low‑latency messaging.

CKafkaIdempotenceKafka
0 likes · 18 min read
Mastering TDMQ CKafka Production: Configurations, Partitioning, and Reliable Messaging
Mike Chen's Internet Architecture
Mike Chen's Internet Architecture
Jul 24, 2025 · Big Data

Boost Kafka Consumer Throughput: Multi‑Threading, Consumer Groups & Config Tuning

This guide explains why Kafka consumer throughput matters, then details practical techniques—including multi‑threaded consumption, scaling with consumer groups, client parameter tuning, and batch processing—to dramatically increase throughput while maintaining reliability in high‑concurrency, large‑scale data pipelines.

ConsumerKafkahigh throughput
0 likes · 5 min read
Boost Kafka Consumer Throughput: Multi‑Threading, Consumer Groups & Config Tuning
Lakehouse Research Base
Lakehouse Research Base
Jul 24, 2025 · Big Data

Two Solutions for ElasticSearch Wildcard Index Collection: Flink ETL Outperforms Logstash/Kafka Pipeline

The author compares two architectures for collecting ElasticSearch wildcard index data when DataX lacks support: a Logstash-Kafka-MaxCompute pipeline with existing tools but longer chain, and a simpler Flink ETL approach writing directly to MaxCompute, concluding Flink is preferable due to lower cost and operational complexity.

DataXETLElasticsearch
0 likes · 3 min read
Two Solutions for ElasticSearch Wildcard Index Collection: Flink ETL Outperforms Logstash/Kafka Pipeline
macrozheng
macrozheng
Jul 17, 2025 · Backend Development

How We Resolved Repeated Kafka Message Backlogs in a High‑Traffic Restaurant System

This article recounts a series of Kafka message backlog incidents in a restaurant ordering system and explains how targeted optimizations—batch database queries, index tuning, data archiving, and thread‑pool scaling—eliminated the delays and restored reliable kitchen display performance.

Backend DevelopmentKafkaMySQL
0 likes · 12 min read
How We Resolved Repeated Kafka Message Backlogs in a High‑Traffic Restaurant System
Architect
Architect
Jul 16, 2025 · Backend Development

Mastering Spring Transaction Hooks: Async Kafka Logging After Commit

This article explains how to use Spring's TransactionSynchronizationManager to detect active transactions and reliably send Kafka messages either immediately or after transaction commit, illustrated with a payment‑system use case and complete code examples.

@AsyncJavaKafka
0 likes · 10 min read
Mastering Spring Transaction Hooks: Async Kafka Logging After Commit
Mike Chen's Internet Architecture
Mike Chen's Internet Architecture
Jul 14, 2025 · Backend Development

Kafka Producer Idempotency: PID, Sequence Numbers, and Broker Deduplication

Kafka ensures that a producer’s repeated message sends, caused by network glitches or broker failures, result in only one persisted record per partition by using a unique Producer ID, monotonically increasing sequence numbers, and broker-side tracking of the latest committed sequence for each PID‑partition pair.

BackendKafkamessage deduplication
0 likes · 5 min read
Kafka Producer Idempotency: PID, Sequence Numbers, and Broker Deduplication
Mike Chen's Internet Architecture
Mike Chen's Internet Architecture
Jul 9, 2025 · Backend Development

How to Prevent Duplicate Consumption in Kafka: Practical Strategies

This article explains why Kafka’s at‑least‑once delivery can cause duplicate message processing, outlines the business risks of such duplicates, and presents four practical solutions—including idempotent design, manual offset commits, exactly‑once semantics, and dead‑letter queues—to ensure reliable consumption.

Exactly-OnceKafkadead letter queue
0 likes · 6 min read
How to Prevent Duplicate Consumption in Kafka: Practical Strategies
Mike Chen's Internet Architecture
Mike Chen's Internet Architecture
Jul 7, 2025 · Big Data

How Kafka Achieves Billion-Message Throughput: Sequential Disk Writes, Page Cache, and Zero‑Copy

This article explains how Kafka sustains massive traffic by writing logs sequentially to disk, leveraging the operating system’s page cache for fast in‑memory writes, employing zero‑copy techniques like sendfile to avoid user‑space copying, and batching messages to reduce network overhead, thereby delivering high‑throughput, low‑latency streaming.

Kafkabatchinghigh throughput
0 likes · 4 min read
How Kafka Achieves Billion-Message Throughput: Sequential Disk Writes, Page Cache, and Zero‑Copy
FunTester
FunTester
Jul 5, 2025 · Big Data

Master Kafka: Core Concepts and Performance Testing Strategies

This article explains Kafka’s high‑performance distributed streaming architecture, key components such as topics, partitions, producers, consumers, brokers, offsets, and ZooKeeper, and provides step‑by‑step workflows for producers and consumers along with performance‑testing tips and Maven setup.

JavaKafkaStreaming
0 likes · 9 min read
Master Kafka: Core Concepts and Performance Testing Strategies
Big Data Tech Team
Big Data Tech Team
Jul 3, 2025 · Big Data

Master Kafka: A Complete Learning Roadmap from Basics to Advanced Projects

This guide presents a step‑by‑step Kafka learning roadmap covering core concepts, architecture, configuration, monitoring tools, practical project ideas, advanced components like Streams and KSQL, plus code samples and resource recommendations to help beginners become proficient in real‑time data streaming.

Code ExamplesKafkaStreaming
0 likes · 14 min read
Master Kafka: A Complete Learning Roadmap from Basics to Advanced Projects
dbaplus Community
dbaplus Community
Jun 25, 2025 · Operations

How We Boosted Kafka Production Capacity by 35% with Simple Compression Tweaks

Facing petabyte‑scale log traffic, the Qunar team identified low compression rates in their Kafka‑Filebeat pipeline as the main bottleneck and, through systematic tuning of batch size, memory queues, and round‑robin settings, achieved a 35% reduction in traffic and a 30‑42% drop in request volume while raising per‑minute throughput by 35%.

BackendCompressionFilebeat
0 likes · 10 min read
How We Boosted Kafka Production Capacity by 35% with Simple Compression Tweaks
Code Ape Tech Column
Code Ape Tech Column
Jun 20, 2025 · Backend Development

Mastering Asynchronous Processing in Java: Build a Robust SDK with Spring

This article introduces a generic asynchronous processing SDK for Java, explaining its purpose, advantages, underlying principles, component choices such as Kafka, XXL‑Job, MySQL, and Spring AOP, and provides detailed configuration, database schema, usage examples, and best‑practice guidelines to achieve reliable, non‑blocking execution and eventual consistency in backend systems.

BackendKafkaSDK
0 likes · 9 min read
Mastering Asynchronous Processing in Java: Build a Robust SDK with Spring
Su San Talks Tech
Su San Talks Tech
Jun 20, 2025 · Backend Development

Why Environment Setup Is the Real Pain in Spring Boot Projects (And How to Fix It)

This article walks through the common frustrations of IDE and environment setup for Spring Boot projects, explains version compatibility between Spring Cloud, Spring Boot, and Kafka, demonstrates how Maven and auto‑configuration simplify dependencies, and provides ready‑to‑use code snippets and useful tooling recommendations for backend developers.

IDE setupKafkaMaven
0 likes · 11 min read
Why Environment Setup Is the Real Pain in Spring Boot Projects (And How to Fix It)
Senior Tony
Senior Tony
Jun 18, 2025 · Backend Development

Mastering High‑Concurrency Account Balance Deductions with Kafka, Sharding, and Parallel Processing

This article examines the challenges of high‑concurrency write scenarios in ad‑billing systems, explains why row‑level locking becomes a bottleneck, and presents a step‑by‑step solution using Kafka for asynchronous peak shaving, parallel consumer processing, database sharding, hotspot dispersion, and batch deduction to achieve reliable real‑time balance updates.

Kafkaaccount balance deductionhigh concurrency
0 likes · 8 min read
Mastering High‑Concurrency Account Balance Deductions with Kafka, Sharding, and Parallel Processing
Architect
Architect
Jun 6, 2025 · Backend Development

How to Prevent Duplicate Messages in Kafka and Pulsar: A Practical Guide

This article explains the three message delivery semantics, the common causes of duplicate messages in queue systems, and presents concrete producer‑side, broker‑side, and consumer‑side deduplication techniques for Kafka and Pulsar, including code samples and best‑practice recommendations.

BackendIdempotenceKafka
0 likes · 9 min read
How to Prevent Duplicate Messages in Kafka and Pulsar: A Practical Guide
Instant Consumer Technology Team
Instant Consumer Technology Team
Jun 5, 2025 · Big Data

Mastering Kafka in Production: Boost Throughput, Ensure Reliability, and Avoid Data Loss

This article shares practical Kafka production insights, covering architecture overview, producer throughput tuning, message loss prevention, broker and consumer configurations, duplicate consumption avoidance, backlog mitigation, ordering guarantees, and the mechanics of consumer group rebalancing, helping engineers build stable, high‑performance streaming pipelines.

Consumer GroupKafkabig data
0 likes · 15 min read
Mastering Kafka in Production: Boost Throughput, Ensure Reliability, and Avoid Data Loss
Mike Chen's Internet Architecture
Mike Chen's Internet Architecture
Jun 3, 2025 · Big Data

Kafka High-Concurrency Core Design Explained

This article explains how Kafka achieves high concurrency through its distributed broker cluster, partitioned topics, sequential log writes, message compression, asynchronous producer mechanisms, and OS page‑cache techniques, illustrating the combined architectural and performance optimizations that enable massive throughput.

Distributed ArchitectureKafkaasynchronous producer
0 likes · 4 min read
Kafka High-Concurrency Core Design Explained
Full-Stack Internet Architecture
Full-Stack Internet Architecture
May 27, 2025 · Big Data

Understanding Event Streaming in Kafka: Core Concepts, Architecture, and Use Cases

This article explains Kafka's event streaming concept, detailing events and streams, core components such as producers, topics, partitions, consumers, persistence, and typical real‑time data pipeline, event‑driven architecture, stream processing, and log aggregation use cases, highlighting its role as a foundational big‑data infrastructure.

Event StreamingKafkamessage queues
0 likes · 7 min read
Understanding Event Streaming in Kafka: Core Concepts, Architecture, and Use Cases
Java Captain
Java Captain
May 23, 2025 · Backend Development

Common Causes of Kafka Message Loss and Mitigation Strategies

This article examines the typical reasons Kafka messages are lost across producers, brokers, and consumers, and provides detailed configuration recommendations and best‑practice solutions to significantly reduce the risk of data loss in distributed streaming systems.

ConfigurationConsumerKafka
0 likes · 15 min read
Common Causes of Kafka Message Loss and Mitigation Strategies
Su San Talks Tech
Su San Talks Tech
May 23, 2025 · Backend Development

Kafka vs RabbitMQ vs RocketMQ vs ActiveMQ: Key Differences and How to Choose

This article explains the fundamentals, architectures, key concepts, advantages, and disadvantages of Kafka, RabbitMQ, RocketMQ, and ActiveMQ, and provides practical guidance on selecting the most suitable message queue for various application scenarios such as high‑throughput logging, financial transactions, or small‑scale services.

KafkaRabbitMQRocketMQ
0 likes · 19 min read
Kafka vs RabbitMQ vs RocketMQ vs ActiveMQ: Key Differences and How to Choose
Mike Chen's Internet Architecture
Mike Chen's Internet Architecture
May 22, 2025 · Big Data

Optimizing Kafka Producer for High Throughput: Batching, Asynchronous Sending, Compression, and Concurrency

This article details practical techniques for boosting Kafka producer throughput, covering batch sending optimization, linger.ms tuning, asynchronous send handling, compression options, and concurrent sending strategies, while providing code examples and performance considerations for high‑throughput streaming applications.

CompressionKafkaasynchronous
0 likes · 4 min read
Optimizing Kafka Producer for High Throughput: Batching, Asynchronous Sending, Compression, and Concurrency
Full-Stack Internet Architecture
Full-Stack Internet Architecture
May 20, 2025 · Big Data

Why Learn Kafka? Core Benefits, Use Cases, and a Summary

This article explains why Kafka is widely adopted by top companies, outlines its high throughput, scalability, and durability, and describes key real‑time data pipeline, stream processing, and big‑data integration scenarios, concluding that mastering Kafka is essential for modern backend and data engineering roles.

Kafkadata engineeringreal-time processing
0 likes · 4 min read
Why Learn Kafka? Core Benefits, Use Cases, and a Summary
Code Ape Tech Column
Code Ape Tech Column
May 19, 2025 · Backend Development

Mastering CQRS in Spring Microservices: From Basics to Event Sourcing

This article explains the CQRS architectural pattern, its benefits and pitfalls, and provides a step‑by‑step guide to implementing CQRS with Spring Boot, Axon, and Kafka, including command and query handling, event sourcing, and practical considerations for microservice architectures.

AxonCQRSEvent Sourcing
0 likes · 13 min read
Mastering CQRS in Spring Microservices: From Basics to Event Sourcing
Selected Java Interview Questions
Selected Java Interview Questions
May 15, 2025 · Backend Development

Six Common Approaches to Synchronize MySQL Data to Elasticsearch

This article reviews six mainstream solutions for keeping MySQL and Elasticsearch in sync—including synchronous double‑write, asynchronous MQ‑based double‑write, Logstash polling, Canal binlog listening, DataX batch migration, and Flink stream processing—detailing their scenarios, advantages, drawbacks, and practical code examples to guide optimal technical selection.

CanalElasticsearchFlink
0 likes · 8 min read
Six Common Approaches to Synchronize MySQL Data to Elasticsearch
Airbnb Technology Team
Airbnb Technology Team
May 13, 2025 · Backend Development

How Airbnb Scaled Its Postcard Generation System for Group Travel

Airbnb's media team engineered a scalable postcard generation pipeline that matches destinations, handles localized text layout, and delivers high‑performance images across iOS, Android, Web, and external messaging platforms using flexible templates, async processing, and pre‑generation strategies.

AirbnbKafkaasynchronous-processing
0 likes · 11 min read
How Airbnb Scaled Its Postcard Generation System for Group Travel
Code Ape Tech Column
Code Ape Tech Column
May 9, 2025 · Databases

Efficient Strategies for Importing One Billion Records into MySQL

This article explains how to import 1 billion 1 KB log records stored in HDFS or S3 into MySQL by analyzing single‑table limits, using batch inserts, choosing storage engines, sharding, optimizing file‑reading methods, and coordinating distributed tasks with Redis, Redisson, and Zookeeper to ensure ordered, reliable, and high‑throughput data loading.

Batch InsertKafkaMySQL
0 likes · 19 min read
Efficient Strategies for Importing One Billion Records into MySQL
Raymond Ops
Raymond Ops
May 7, 2025 · Operations

How Kafka Elects Leaders and Distributes Partitions: Inside the Mechanics

Kafka’s internal mechanisms for leader election, partition assignment, and file storage are explained, covering how the Controller uses ZooKeeper, the ISR-based leader selection process, partition distribution strategies, segment file structures, and the evolution of offset management from Zookeeper to the __consumer_offsets topic.

KafkaLeader ElectionOffsets
0 likes · 5 min read
How Kafka Elects Leaders and Distributes Partitions: Inside the Mechanics
Su San Talks Tech
Su San Talks Tech
May 5, 2025 · Big Data

6 Proven Ways to Sync MySQL Data to Elasticsearch – Choose the Right Strategy

This article compares six mainstream MySQL‑to‑Elasticsearch synchronization methods—synchronous double‑write, asynchronous MQ, Logstash polling, Canal binlog listening, DataX batch sync, and Flink streaming—detailing scenarios, code samples, advantages, drawbacks, and practical selection guidance for developers.

CanalElasticsearchFlink
0 likes · 9 min read
6 Proven Ways to Sync MySQL Data to Elasticsearch – Choose the Right Strategy
dbaplus Community
dbaplus Community
Apr 16, 2025 · Backend Development

How Ctrip’s Kafka Gatekeeper Boosts FinOps Data Quality and Automates Cost Governance

This article explains how Ctrip’s hybrid‑cloud FinOps billing system uses a custom Kafka Gatekeeper to detect, locate, and automatically remediate data‑quality issues across dozens of self‑built PaaS services, improving coverage, timeliness, and responsibility attribution while supporting high‑availability deployments.

BackendCloud NativeData Quality
0 likes · 19 min read
How Ctrip’s Kafka Gatekeeper Boosts FinOps Data Quality and Automates Cost Governance
macrozheng
macrozheng
Apr 11, 2025 · Backend Development

How to Guarantee Exactly‑Once Message Consumption in High‑Concurrency Systems

This article explains common causes of duplicate message consumption in high‑traffic systems and presents a three‑layer defense—producer idempotence, broker de‑duplication, and consumer idempotent design—plus monitoring and reconciliation strategies to achieve reliable exactly‑once processing.

Exactly-OnceIdempotenceKafka
0 likes · 7 min read
How to Guarantee Exactly‑Once Message Consumption in High‑Concurrency Systems
Sanyou's Java Diary
Sanyou's Java Diary
Apr 10, 2025 · Backend Development

Why RocketMQ Beats Kafka: Architecture Simplified and Features Amplified

This article explains how RocketMQ, a Chinese‑origin message queue, simplifies Kafka’s architecture while adding powerful features such as tag‑based filtering, transactional messaging, delayed and dead‑letter queues, and a unified commit‑log storage model, making delayed processing and high‑throughput scenarios easier to implement.

ArchitectureKafkaRocketMQ
0 likes · 10 min read
Why RocketMQ Beats Kafka: Architecture Simplified and Features Amplified
Architecture Digest
Architecture Digest
Apr 6, 2025 · Backend Development

Design and Implementation of a General‑Purpose Asynchronous Processing SDK for Backend Systems

This article introduces a reusable asynchronous processing SDK built on Spring, Kafka, and MySQL that leverages @AsyncExec annotations, transactional event listeners, and configurable thread pools to ensure reliable, non‑blocking execution, data consistency, and fault‑tolerant handling of business logic in backend applications.

@AsyncBackendKafka
0 likes · 8 min read
Design and Implementation of a General‑Purpose Asynchronous Processing SDK for Backend Systems
Ma Wei Says
Ma Wei Says
Mar 30, 2025 · Fundamentals

How Kafka 4.0’s KRaft Replaces ZooKeeper with Raft Consensus

Kafka 4.0 introduces KRaft, a ZooKeeper‑free metadata layer built on the Raft consensus algorithm, detailing role transitions, leader election, log replication, controller and broker responsibilities, and fault‑tolerance mechanisms, enabling a more scalable and self‑managed architecture for large‑scale distributed streaming.

Consensus AlgorithmKRaftKafka
0 likes · 13 min read
How Kafka 4.0’s KRaft Replaces ZooKeeper with Raft Consensus
Ma Wei Says
Ma Wei Says
Mar 28, 2025 · Backend Development

Choosing the Right Message Queue: Kafka vs RocketMQ vs RabbitMQ Explained

This article compares Kafka, RocketMQ, and RabbitMQ, detailing their architectures, performance characteristics, strengths, and ideal use‑cases to help engineers select the most suitable message‑queue solution for high‑throughput, fault‑tolerant, and real‑time processing scenarios.

Event StreamingKafkaRabbitMQ
0 likes · 11 min read
Choosing the Right Message Queue: Kafka vs RocketMQ vs RabbitMQ Explained
Didi Tech
Didi Tech
Mar 27, 2025 · Operations

Performance Optimization and Architecture of iLogTail for High‑Scale Log Collection

Didi replaced its legacy agent with Alibaba’s open‑source iLogTail, re‑architected it to use a shared thread‑pool and SIMD‑accelerated parsing, rewrote critical plugins in C++ and added robust Kafka retry logic, achieving over twice the throughput while cutting CPU usage by more than half and maintaining near‑zero latency at massive scale.

C++KafkaLog Collection
0 likes · 10 min read
Performance Optimization and Architecture of iLogTail for High‑Scale Log Collection
Linux Ops Smart Journey
Linux Ops Smart Journey
Mar 27, 2025 · Cloud Native

Stream Kubernetes Events to Elasticsearch with Kafka & Logstash

This guide walks you through deploying the Kubernetes Event Exporter, packaging its Helm chart and Docker image, configuring Kafka TLS secrets, setting up Logstash to ingest events and forward them to Elasticsearch, creating an index template, and verifying the end‑to‑end pipeline.

ElasticsearchEvent ExporterHelm
0 likes · 10 min read
Stream Kubernetes Events to Elasticsearch with Kafka & Logstash
iQIYI Technical Product Team
iQIYI Technical Product Team
Mar 27, 2025 · Big Data

Cost‑Effective Real‑Time Data Warehouse 2.0: Migrating from Kafka to Iceberg

iQIYI transformed its real‑time data warehouse by replacing a costly Kafka‑based Lambda stack with a unified stream‑batch Iceberg lake, cutting storage expenses by 90%, halving compute costs, extending data retention, and delivering minute‑level freshness for 90% of use cases while preserving second‑level processing where needed.

FlinkKafkaReal-time Data Warehouse
0 likes · 11 min read
Cost‑Effective Real‑Time Data Warehouse 2.0: Migrating from Kafka to Iceberg
Su San Talks Tech
Su San Talks Tech
Mar 27, 2025 · Operations

How to Ensure Data Consistency in Message Queues: 10 Hard‑Earned Lessons

This article explores why message queues can lose consistency, presents concrete solutions such as transactional two‑phase commits, persistence settings, replica configurations, unique IDs, idempotent designs, and dead‑letter queues, and shares ten practical lessons drawn from real‑world incidents.

KafkaRabbitMQTransactional Messaging
0 likes · 12 min read
How to Ensure Data Consistency in Message Queues: 10 Hard‑Earned Lessons
Big Data Technology Architecture
Big Data Technology Architecture
Mar 25, 2025 · Big Data

Kafka 4.0 Release: KRaft Architecture, Consumer Group Optimizations, and New Queue Features

Kafka 4.0 marks a milestone release that replaces ZooKeeper with the KRaft consensus engine, improves scalability and performance, introduces a server‑side consumer‑group protocol, adds shared‑group queue capabilities, and updates Java requirements and documentation, delivering a more robust and flexible streaming platform.

Distributed StreamingJava11KRaft
0 likes · 6 min read
Kafka 4.0 Release: KRaft Architecture, Consumer Group Optimizations, and New Queue Features