Skip to content
Tagged articles

Kafka

500 articles · Page 3 of 5
Mike Chen's Internet Architecture
Mike Chen's Internet Architecture
Dec 3, 2025 · Big Data

Kafka High‑Throughput Tricks: Sequential Writes, Zero‑Copy, Partitioning

The article explains how Kafka achieves high throughput by writing messages sequentially to disk, leveraging OS page cache and zero‑copy system calls, using partitioned topics for parallelism, batching and compressing records on both producer and broker sides, and employing asynchronous replication with configurable persistence strategies.

KafkaPartitioningbatching
0 likes · 4 min read
Kafka High‑Throughput Tricks: Sequential Writes, Zero‑Copy, Partitioning
Code Wrench
Code Wrench
Nov 22, 2025 · Backend Development

Build a Production-Ready Rule Engine with Gray Release Using Go, Kafka, and Redis

Learn how to design and implement a ready-to-use rule engine combined with a gray release system using Golang, Kafka, Redis, and CEL, complete with Docker‑compose deployment, edge execution, token‑bucket throttling, and webhook actions, plus full source code for a production‑grade marketing strategy platform.

CELGoGray Release
0 likes · 9 min read
Build a Production-Ready Rule Engine with Gray Release Using Go, Kafka, and Redis
dbaplus Community
dbaplus Community
Nov 18, 2025 · Backend Development

How to Guarantee 100% No Message Loss in Distributed MQ Systems

Ensuring that messages never disappear in a distributed MQ system requires a three‑pronged strategy covering production, storage, and consumption, with proper ACK configurations, local message tables, replication settings, and manual offset commits to achieve reliable, at‑least‑once processing without data loss.

BackendKafkaMQ
0 likes · 11 min read
How to Guarantee 100% No Message Loss in Distributed MQ Systems
MaGe Linux Operations
MaGe Linux Operations
Nov 18, 2025 · Big Data

Zero‑Data‑Loss Kafka Cluster Scaling: Complete Step‑by‑Step Guide

This comprehensive guide explains how to safely expand a Kafka cluster without data loss by covering applicable scenarios, pre‑conditions, anti‑pattern warnings, environment matrices, a detailed checklist, step‑by‑step Linux commands for broker preparation, partition‑rebalancing plan generation, throttled execution, real‑time monitoring, verification, rollback procedures, backup strategies, performance testing, common troubleshooting, FAQs and best‑practice scripts, all illustrated with code snippets and practical examples.

Cluster ScalingKafkaLinux
0 likes · 47 min read
Zero‑Data‑Loss Kafka Cluster Scaling: Complete Step‑by‑Step Guide
Ray's Galactic Tech
Ray's Galactic Tech
Nov 17, 2025 · Backend Development

Kafka vs RabbitMQ vs RocketMQ: Which Message Broker Fits Your Use Case?

This article provides an in‑depth, multi‑dimensional comparison of Apache Kafka, RabbitMQ, and Apache RocketMQ—covering design philosophy, performance, reliability, features, ecosystem, operations, and typical scenarios—to help you choose the right message middleware for your architecture.

KafkaRabbitMQRocketMQ
0 likes · 6 min read
Kafka vs RabbitMQ vs RocketMQ: Which Message Broker Fits Your Use Case?
Tech Freedom Circle
Tech Freedom Circle
Nov 15, 2025 · Databases

How to Prevent Order Loss in a 100k TPS Flash Sale When the Master DB Crashes – 5 Practical Solutions

The article dissects a high‑traffic flash‑sale interview question—how to guarantee zero order loss at 100,000 TPS when the master MySQL instance fails—by explaining the underlying performance‑consistency conflict, the three skills interviewers assess, and presenting five concrete, code‑driven solutions ranging from MySQL parameter tuning to semi‑sync replication, local message tables, group replication, and Redis‑Kafka traffic shaping.

Group ReplicationKafkaMySQL
0 likes · 28 min read
How to Prevent Order Loss in a 100k TPS Flash Sale When the Master DB Crashes – 5 Practical Solutions
mikechen
mikechen
Nov 13, 2025 · Backend Development

How to Diagnose and Resolve Kafka Message Backlog Issues

This article explains what Kafka message backlog is, outlines the main reasons it occurs—such as producer speed outpacing consumers, slow consumer processing, and downstream bottlenecks—and provides practical steps for producer throttling, consumer scaling and logic improvements, and Kafka cluster enhancements to eliminate the backlog.

Backend DevelopmentKafkaMessage Backlog
0 likes · 6 min read
How to Diagnose and Resolve Kafka Message Backlog Issues
Ray's Galactic Tech
Ray's Galactic Tech
Nov 9, 2025 · Backend Development

Hybrid Push‑Pull Timeline Architecture: Scaling Social Feeds for Billions

To serve billions of users with real‑time timelines, modern social platforms combine push‑based delivery for regular users and pull‑based retrieval for high‑profile accounts, employing hot‑cold separation, Kafka fan‑out, Redis caching, and scalable storage strategies to balance write and read loads.

KafkaPush‑PullRedis
0 likes · 9 min read
Hybrid Push‑Pull Timeline Architecture: Scaling Social Feeds for Billions
Ops Community
Ops Community
Nov 6, 2025 · Big Data

Zero Data Loss Kafka Cluster Scaling: From 3 to 10 Nodes – A Complete Guide

This comprehensive guide walks you through expanding or shrinking a production‑grade Kafka cluster—covering prerequisites, anti‑pattern warnings, environment matrices, step‑by‑step expansion and contraction procedures, partition rebalancing principles, monitoring, best practices, and troubleshooting—to ensure zero data loss during scaling.

Cluster ScalingKafkaPartition Rebalancing
0 likes · 27 min read
Zero Data Loss Kafka Cluster Scaling: From 3 to 10 Nodes – A Complete Guide
DataFunTalk
DataFunTalk
Nov 6, 2025 · Cloud Native

How Tencent Music Cut Kafka Costs by 50% with Cloud‑Native AutoMQ

Tencent Music migrated its massive Kafka streaming infrastructure to the cloud‑native AutoMQ platform, slashing operational costs by over half, achieving second‑level partition migration, and dramatically improving scaling efficiency while maintaining high‑throughput, low‑latency data processing for its music services.

AutoMQData StreamingKafka
0 likes · 16 min read
How Tencent Music Cut Kafka Costs by 50% with Cloud‑Native AutoMQ
Top Architect
Top Architect
Oct 31, 2025 · Backend Development

Mastering Message Queues: A Deep Dive into RabbitMQ, RocketMQ, and Kafka

This comprehensive guide explains the core components, exchange types, TTL, confirm mechanisms, consumer ACK/NACK, dead‑letter queues, and high‑availability features of RabbitMQ, RocketMQ, and Kafka, while also covering load balancing, ordering, transaction handling, and best practices for reliable message delivery.

Backend DevelopmentKafkaMessage Queue
0 likes · 32 min read
Mastering Message Queues: A Deep Dive into RabbitMQ, RocketMQ, and Kafka
Big Data Technology & Architecture
Big Data Technology & Architecture
Oct 30, 2025 · Backend Development

What’s New in Apache Kafka 4.1? Core Features and Architecture Changes Explained

Apache Kafka 4.1.0 introduces native queue semantics, a new Streams rebalancing protocol, multi‑version Connect plugins, a revamped consumer‑group protocol, enhanced transaction safety, and numerous client, monitoring, and security improvements, offering a comprehensive upgrade over the 4.0 release.

Kafkadistributed-systemskafka-4.1
0 likes · 6 min read
What’s New in Apache Kafka 4.1? Core Features and Architecture Changes Explained
Su San Talks Tech
Su San Talks Tech
Oct 28, 2025 · Backend Development

How to Prevent MQ Message Loss: 5 Proven Strategies for Reliable Messaging

Discover the three stages where MQ messages can be lost and explore five practical solutions—including producer confirmations, message persistence, consumer acknowledgments, transactional messaging, and retry with dead‑letter queues—complete with code examples and guidance on selecting the right approach for different scenarios.

Dead Letter QueueKafkaMessage Queue
0 likes · 14 min read
How to Prevent MQ Message Loss: 5 Proven Strategies for Reliable Messaging
Shepherd Advanced Notes
Shepherd Advanced Notes
Oct 24, 2025 · Backend Development

Why Choose Spring Boot + DelayQueue for a Custom Distributed Delayed-Task Queue?

The article systematically analyzes common distributed delayed‑task implementations—Redis ZSet scanning, message‑queue delay features, and Redis key‑expiration listeners—highlighting their pros, cons, and suitable scenarios, then proposes a Spring Boot + DelayQueue component to achieve precise timing, dynamic delays, and robust coordination.

DelayQueueKafkaRedis
0 likes · 11 min read
Why Choose Spring Boot + DelayQueue for a Custom Distributed Delayed-Task Queue?
Huolala Tech
Huolala Tech
Oct 22, 2025 · Backend Development

Scaling Real‑Time Reconciliation with Dynamic Kafka Consumer Clusters

To ensure fund safety and robust operations, the team built a real‑time reconciliation platform that leverages Kafka, and after encountering scaling bottlenecks with a static consumer model, they implemented a dynamic, partition‑level, weighted load‑balancing consumer cluster that supports automatic scaling and high‑throughput processing.

Backend ArchitectureDynamic ScalingKafka
0 likes · 15 min read
Scaling Real‑Time Reconciliation with Dynamic Kafka Consumer Clusters
Architect Chen
Architect Chen
Oct 22, 2025 · Big Data

How to Eliminate Kafka Message Backlog with Practical Optimizations

This guide presents concrete techniques for improving Kafka consumer and producer performance, scaling clusters, tuning broker settings, and designing asynchronous buffering layers to prevent message accumulation and boost overall throughput.

KafkaPerformance Optimizationbig data
0 likes · 5 min read
How to Eliminate Kafka Message Backlog with Practical Optimizations
Raymond Ops
Raymond Ops
Oct 21, 2025 · Big Data

Deep Dive into Kafka Architecture: Topics, Partitions, and Reliable Data Pipelines

This article explains Kafka’s core concepts—including topics, partitions, log segmentation, indexing, and acknowledgment mechanisms—then provides a step‑by‑step guide to deploy a Zookeeper‑Kafka cluster integrated with Filebeat, Logstash, and the ELK stack for reliable log collection and analysis.

KafkaZooKeeperbig data
0 likes · 11 min read
Deep Dive into Kafka Architecture: Topics, Partitions, and Reliable Data Pipelines
Code Wrench
Code Wrench
Oct 18, 2025 · Backend Development

Master High‑Performance Queues in Go: Kafka, RabbitMQ & Redis Compared

This article explains how to build a high‑throughput, low‑latency, and scalable queue system in Go by leveraging Kafka, RabbitMQ, and Redis, covering core concepts, practical code examples, performance optimizations, and guidance on choosing the right solution for different workloads.

ConcurrencyGoKafka
0 likes · 11 min read
Master High‑Performance Queues in Go: Kafka, RabbitMQ & Redis Compared
Programmer XiaoFu
Programmer XiaoFu
Oct 16, 2025 · Operations

Stop Guessing: Kafka Message Backlog, Duplicates, and Loss Are Usually Caused by Rebalance

Kafka consumer issues such as message backlog, duplicate processing, and data loss often stem from consumer group rebalances triggered by changes in consumer count, partition count, subscription topics, or heartbeat and poll timeouts, and can be mitigated by tuning timeout settings, managing offset commits, and using sticky partition assignment.

Consumer GroupKafkaMessage Duplication
0 likes · 13 min read
Stop Guessing: Kafka Message Backlog, Duplicates, and Loss Are Usually Caused by Rebalance
Ubiquitous Tech
Ubiquitous Tech
Oct 15, 2025 · Backend Development

Java Interview Simulation: Solving the Architecture Challenges of a Hotel Pricing System

This article walks through a detailed Java interview simulation where the candidate designs a high‑throughput hotel pricing service, covering business decomposition, Kafka event‑driven pipelines, cache‑aside consistency, HikariCP tuning, fault‑tolerant microservices, multi‑region disaster recovery, and AI recommendation gray‑release strategies.

IdempotencyJavaKafka
0 likes · 43 min read
Java Interview Simulation: Solving the Architecture Challenges of a Hotel Pricing System
StarRocks
StarRocks
Oct 14, 2025 · Big Data

How Ctrip Scaled UBT Analytics by Migrating from ClickHouse to StarRocks

Ctrip's User Behavior Tracking (UBT) system, handling 30 TB of daily data, moved from ClickHouse to StarRocks' compute‑storage separated architecture, cutting average query latency from 1.4 seconds to 203 ms, halving storage, reducing nodes from 50 to 40, and boosting write throughput to 3 million rows per second.

ClickHouseFlinkKafka
0 likes · 15 min read
How Ctrip Scaled UBT Analytics by Migrating from ClickHouse to StarRocks
Smart Sea Tide
Smart Sea Tide
Oct 13, 2025 · Big Data

How Zhihu Evolved Its Real-Time Data Warehouse Architecture

The article details Zhihu's step‑by‑step evolution from a Spark‑Streaming‑based real‑time data warehouse (v1.0) to a Flink‑Streaming SQL‑driven platform (v2.0), explaining design choices, Lambda architecture implementation, scalability challenges, and future automation plans.

DruidFlinkHBase
0 likes · 22 min read
How Zhihu Evolved Its Real-Time Data Warehouse Architecture
Efficient Ops
Efficient Ops
Oct 12, 2025 · Information Security

What Exposed the AI Companion Apps? A Deep Dive into the Massive Data Leak

A recent breach of the AI companion apps Chattee and GiMe Chat exposed over 40 million private conversations, hundreds of thousands of media files, and transaction records due to publicly accessible Kafka brokers lacking any authentication, putting user privacy and security at serious risk.

AI securityKafkadata breach
0 likes · 3 min read
What Exposed the AI Companion Apps? A Deep Dive into the Massive Data Leak
Architect
Architect
Oct 11, 2025 · Backend Development

Build a Scalable Spring Cloud Recommendation System: Architecture, Code, and Fault Tolerance

This article walks through the design and implementation of a simplified TikTok‑style recommendation system using Spring Cloud microservices, covering business requirements, service decomposition, Eureka registration, Feign clients, Redis and Kafka integration, circuit‑breaker fallback, and end‑to‑end testing.

JavaKafkaRecommendation System
0 likes · 30 min read
Build a Scalable Spring Cloud Recommendation System: Architecture, Code, and Fault Tolerance
ITPUB
ITPUB
Oct 5, 2025 · Backend Development

How to Clear a 10‑Million‑Message Queue in 5 Hours: A Five‑Step Rescue Plan

When a flash‑sale causes a 10 million‑message backlog and consumers only process 200 messages per second, this guide shows a five‑step, 5‑hour strategy—horizontal scaling, message downgrade, flow control, temporary dump, and parallel blasting—to restore throughput and prevent system collapse.

KafkaPerformance Optimizationdistributed systems
0 likes · 6 min read
How to Clear a 10‑Million‑Message Queue in 5 Hours: A Five‑Step Rescue Plan
Lakehouse Research Base
Lakehouse Research Base
Oct 3, 2025 · Operations

100 Advanced Linux Interview Questions for Big Data Operations

This comprehensive guide presents 100 advanced Linux interview questions tailored for big data operations, covering command-line mastery, system performance tuning, cluster configuration for Hadoop, Kafka, Spark, and Flink, complex troubleshooting scenarios, automation with Ansible and Prometheus, security hardening, and containerized deployments on Kubernetes.

HadoopKafkaLinux
0 likes · 45 min read
100 Advanced Linux Interview Questions for Big Data Operations
ITPUB
ITPUB
Oct 3, 2025 · Big Data

How Qunar Travel Cut 2000 CPU Cores by Optimizing Kafka Production

This case study details how Qunar Travel's engineering team analyzed Kafka production bottlenecks during peak traffic, added targeted monitoring, tuned thread and batch parameters, and validated the changes through gray‑scale tests, ultimately saving about 2000 CPU cores across three clusters while reducing request volume and improving network and disk utilization.

CPU SavingsKafkaPerformance Optimization
0 likes · 14 min read
How Qunar Travel Cut 2000 CPU Cores by Optimizing Kafka Production
Cognitive Technology Team
Cognitive Technology Team
Oct 3, 2025 · Backend Development

Mastering Kafka Consumer Rebalance: Strategies to Boost Throughput and Stability

This article deeply explores Kafka consumer group rebalance mechanisms, identifies performance pitfalls of frequent rebalances, and provides a comprehensive set of configuration tweaks, assignment strategies, batch processing techniques, and monitoring practices to achieve a more stable and high‑throughput Kafka consumer system.

BackendConsumerKafka
0 likes · 16 min read
Mastering Kafka Consumer Rebalance: Strategies to Boost Throughput and Stability
Lakehouse Research Base
Lakehouse Research Base
Oct 2, 2025 · Operations

100 Linux Interview Questions for Big Data Operations

This comprehensive guide presents 100 Linux interview questions tailored for big data operations, covering basic commands, permissions, process management, networking, log processing, shell scripting, and cluster-specific operations with detailed answers and practical explanations.

HadoopInterview QuestionsKafka
0 likes · 41 min read
100 Linux Interview Questions for Big Data Operations
IT Services Circle
IT Services Circle
Oct 1, 2025 · Backend Development

Designing a Fast, Reliable, Cost‑Effective Like System for High‑Traffic Apps

This article breaks down the essential requirements and architecture of a high‑performance like system, covering fast response, data consistency, scalability under traffic spikes, and cost‑efficient resource use, while detailing the technical stack, caching strategies, async persistence, and practical optimizations.

KafkaPerformancebackend design
0 likes · 17 min read
Designing a Fast, Reliable, Cost‑Effective Like System for High‑Traffic Apps
Architecture Digest
Architecture Digest
Sep 23, 2025 · Backend Development

How to Ensure Zero Message Loss in Kafka: Proven Strategies for High‑Reliability Systems

This article explains Kafka's storage architecture, identifies three major message‑loss scenarios across production, storage, and consumption, and provides practical end‑to‑end configurations, detection methods, and business‑level patterns to achieve near‑zero message loss in high‑concurrency distributed systems.

KafkaMessage Queuedata consistency
0 likes · 13 min read
How to Ensure Zero Message Loss in Kafka: Proven Strategies for High‑Reliability Systems
StarRocks
StarRocks
Sep 23, 2025 · Databases

How Zepto Scaled Real‑Time Brand Analytics with StarRocks: From Postgres MVP to Sub‑Second Queries

Zepto transformed its brand‑analytics platform from a Postgres MVP into a production‑grade, sub‑second real‑time analytics solution by adopting StarRocks, redesigning its data pipeline with Databricks, Kafka, and Flink, and choosing a storage‑compute architecture that supports massive joins and rapid insights.

Data PipelineDatabricksFlink
0 likes · 14 min read
How Zepto Scaled Real‑Time Brand Analytics with StarRocks: From Postgres MVP to Sub‑Second Queries
Su San Talks Tech
Su San Talks Tech
Sep 23, 2025 · Backend Development

How to Guarantee 100% Message Delivery with Kafka: Interview‑Ready Strategies

This article dissects Kafka’s storage architecture, identifies loss points in production, storage, and consumption phases, and presents interview‑ready strategies—including acks settings, flush tuning, consumer batch commits, detection via sequence numbers, and transactional messaging—to guarantee virtually 100 % message durability.

Consumer CommitKafkaTransactional Messaging
0 likes · 20 min read
How to Guarantee 100% Message Delivery with Kafka: Interview‑Ready Strategies
Ray's Galactic Tech
Ray's Galactic Tech
Sep 21, 2025 · Cloud Native

How to Deploy a Scalable Kafka Cluster on Bare Metal and Kubernetes

This guide walks you through installing a high‑availability Kafka cluster, covering both bare‑metal/virtual‑machine setups and Kubernetes‑based deployments with Helm, including environment preparation, ZooKeeper configuration, broker settings, Helm chart customization, KRaft mode, and production‑grade tuning recommendations.

HelmKRaftKafka
0 likes · 7 min read
How to Deploy a Scalable Kafka Cluster on Bare Metal and Kubernetes
Su San Talks Tech
Su San Talks Tech
Sep 16, 2025 · Backend Development

Mastering Message Order in Distributed Queues: From Basics to Advanced Strategies

This article explores the fundamentals of message ordering in distributed message queues, explains why ordering is determined by broker arrival, compares global and partial ordering, and presents practical solutions—from single-partition designs to multi-partition hashing, handling data skew, and safe expansion—plus interview tips.

KafkaPartitioningconsistent-hashing
0 likes · 24 min read
Mastering Message Order in Distributed Queues: From Basics to Advanced Strategies
Mike Chen's Internet Architecture
Mike Chen's Internet Architecture
Sep 12, 2025 · Backend Development

How to Prevent Kafka Message Loss: Producer, Broker, and Consumer Strategies

This guide explains comprehensive Kafka message loss prevention techniques, covering producer configurations like acks=all, retries, idempotence, broker settings such as replication factor and min.insync.replicas, consumer handling with manual offset commits, and full‑stack compensation and retry mechanisms with code examples.

Broker SettingsKafkaProducer Configuration
0 likes · 3 min read
How to Prevent Kafka Message Loss: Producer, Broker, and Consumer Strategies
mikechen
mikechen
Sep 11, 2025 · Backend Development

How Does Kafka Guarantee Message Order? Key Mechanisms Explained

This article explains why message ordering is critical in scenarios such as financial transactions, e‑commerce order updates, audit logs, and IoT devices, then details Kafka’s core components—topics, partitions, offsets, producers, brokers, and consumer groups—and describes how single‑partition consumption and consistent key hashing ensure ordered processing while balancing throughput.

BackendConsumer GroupKafka
0 likes · 5 min read
How Does Kafka Guarantee Message Order? Key Mechanisms Explained
Architect Chen
Architect Chen
Sep 10, 2025 · Big Data

How Kafka Achieves Million‑Message Throughput: Sequential Writes, Page Cache, Batching & Zero‑Copy

The article explains how Kafka attains high‑throughput performance by using sequential disk writes, leveraging the OS page cache, employing producer and consumer batching with configurable parameters, and utilizing zero‑copy sendfile to minimize CPU and memory overhead, enabling stable million‑message per second rates.

KafkaSequential Writebatching
0 likes · 5 min read
How Kafka Achieves Million‑Message Throughput: Sequential Writes, Page Cache, Batching & Zero‑Copy
IT Architects Alliance
IT Architects Alliance
Sep 8, 2025 · Cloud Native

Choosing the Right Message Queue: Kafka vs RabbitMQ vs Pulsar – A Practical Guide

This article examines the core evaluation criteria for enterprise message queues and provides a detailed comparison of Kafka, RabbitMQ, and Pulsar—including performance, reliability, scalability, operational complexity, ecosystem maturity, and business fit—offering actionable guidance for selecting the optimal solution in cloud‑native environments.

KafkaPulsarRabbitMQ
0 likes · 9 min read
Choosing the Right Message Queue: Kafka vs RabbitMQ vs Pulsar – A Practical Guide
Ops Community
Ops Community
Sep 8, 2025 · Operations

Mastering Distributed Log Architecture: From Flume to ELK and Beyond

This comprehensive guide walks you through the challenges of large‑scale log collection, real‑time processing, storage optimization, and visualization, detailing practical configurations for Flume, Logstash, Elasticsearch, Kibana, Filebeat, Kafka, Kubernetes, and future AIOps integrations to build a reliable, cost‑effective distributed logging system.

FlumeKafkaelk
0 likes · 24 min read
Mastering Distributed Log Architecture: From Flume to ELK and Beyond
Smart Sea Tide
Smart Sea Tide
Sep 8, 2025 · Big Data

How to Migrate Data from HBase to Kafka: A Reverse Data Flow Guide

This article explains how to move data from an HBase cluster back into Kafka by extracting RowKeys with a MapReduce job, handling massive volumes and lack of partitioning, and reliably writing the records to Kafka while tracking successes and failures.

FirstKeyOnlyFilterHBaseHDFS
0 likes · 9 min read
How to Migrate Data from HBase to Kafka: A Reverse Data Flow Guide
IT Services Circle
IT Services Circle
Sep 6, 2025 · Backend Development

10 Real‑World Scenarios Where Message Queues Transform Your System

This article explores ten practical use‑cases for message queues—covering system decoupling, asynchronous processing, traffic shaping, data synchronization, log collection, broadcast updates, ordered and delayed messages, retry mechanisms, and transactional messaging—illustrated with Java code examples and architectural diagrams.

Backend DevelopmentJavaKafka
0 likes · 17 min read
10 Real‑World Scenarios Where Message Queues Transform Your System
Mike Chen's Internet Architecture
Mike Chen's Internet Architecture
Sep 4, 2025 · Backend Development

How to Diagnose and Fix Kafka Message Backlog Issues

This article explains what Kafka message backlog is, why it harms performance and stability, and provides practical producer‑side throttling and consumer‑side scaling techniques, illustrated with diagrams, to help engineers eliminate backlog and keep their Kafka clusters healthy.

KafkaMessage Backlogconsumer scaling
0 likes · 4 min read
How to Diagnose and Fix Kafka Message Backlog Issues
Tech Freedom Circle
Tech Freedom Circle
Sep 4, 2025 · Backend Development

How to Solve ES Latency in MySQL‑Canal Sync and Indexing Scenarios?

The article dissects the interview question about ES latency in a MySQL‑Canal‑to‑Elasticsearch pipeline, explains the root causes across four system layers, and presents a comprehensive four‑layer optimization, end‑to‑end observability, routing‑based degradation, and a Java‑based LatencyProbe component to measure and control delay.

CanalElasticsearchKafka
0 likes · 17 min read
How to Solve ES Latency in MySQL‑Canal Sync and Indexing Scenarios?
ITPUB
ITPUB
Sep 3, 2025 · Backend Development

How We Boosted Kafka Throughput by 35% with Filebeat Tuning and Compression Tricks

This case study details how a high‑traffic Kafka logging cluster was optimized by analyzing low compression ratios, tuning Filebeat parameters, adjusting memory queues and round‑robin settings, and validating the changes through gray‑scale tests, resulting in up to 35% higher throughput and significant resource savings.

Kafkacompressionfilebeat
0 likes · 10 min read
How We Boosted Kafka Throughput by 35% with Filebeat Tuning and Compression Tricks
php Courses
php Courses
Aug 29, 2025 · Operations

How to Build a Real‑Time PHP Log Event Pipeline for Instant Insights

Learn how to transform PHP logs into real‑time, structured events by implementing a log event pipeline that includes JSON logging, lightweight collectors like Filebeat, streaming platforms such as Kafka or Flink, enrichment, and visualization with Grafana, enabling instant monitoring, alerting, and data‑driven decisions.

FlinkGrafanaKafka
0 likes · 7 min read
How to Build a Real‑Time PHP Log Event Pipeline for Instant Insights
Ubiquitous Tech
Ubiquitous Tech
Aug 26, 2025 · Backend Development

Java Interview Simulation: Solving Sohu’s High‑Throughput Push System

This article walks through a five‑round Java interview simulation where the candidate designs, optimizes, and secures a massive push‑notification system for Sohu, covering Spring Boot high‑concurrency architecture, sharding and cache‑penetration mitigation, Kafka reliability, Kubernetes deployment with gray releases, multi‑region high‑availability, and AI‑driven smart‑push strategies, complete with concrete code snippets and real‑world performance numbers.

KafkaRedisinterview
0 likes · 37 min read
Java Interview Simulation: Solving Sohu’s High‑Throughput Push System
Ubiquitous Tech
Ubiquitous Tech
Aug 24, 2025 · Backend Development

Java Interview Deep Dive: Solving Real‑World Invoice System Challenges

The article walks through a simulated five‑round Java interview where the candidate designs a high‑concurrency invoice‑issuing service, covering gateway architecture, async Kafka processing, cache strategies, JVM thread‑pool and memory tuning, database‑cache coordination, microservice messaging, system design, performance optimization and AI integration, all illustrated with concrete numbers and code snippets.

JVMJavaKafka
0 likes · 35 min read
Java Interview Deep Dive: Solving Real‑World Invoice System Challenges
Su San Talks Tech
Su San Talks Tech
Aug 23, 2025 · Backend Development

How to Build a Robust Asynchronous Processing SDK with Spring and Kafka

This article explains how to design and implement a generic asynchronous processing SDK for Java backend services, covering its purpose, advantages, underlying principles, component choices, database schema, configuration, usage patterns, and best‑practice notes, with code examples and diagrams.

Asynchronous ProcessingJava backendKafka
0 likes · 9 min read
How to Build a Robust Asynchronous Processing SDK with Spring and Kafka
Alibaba Cloud Big Data AI Platform
Alibaba Cloud Big Data AI Platform
Aug 21, 2025 · Big Data

How Hypergryph Built a High‑Performance Real‑Time Analytics Platform with StarRocks

This case study details how Hypergryph leveraged Alibaba Cloud EMR Serverless StarRocks, Flink, and Kafka to replace a ClickHouse data warehouse with a high‑performance, elastic, and easy‑to‑operate real‑time analytics platform that dramatically improved query speed, stability, operational efficiency, and cost for their gaming business.

Data PipelineFlinkKafka
0 likes · 8 min read
How Hypergryph Built a High‑Performance Real‑Time Analytics Platform with StarRocks
Mike Chen's Internet Architecture
Mike Chen's Internet Architecture
Aug 20, 2025 · Backend Development

How Kafka Achieves High‑Performance Storage: Log Segments, Indexes, and Retention

This article explains Kafka's storage architecture, including its partition‑based log files, sequential append writes, log segment management, index files, and configurable time‑ and size‑based retention policies that together enable ultra‑high write throughput while controlling disk usage.

KafkaRetention PolicySequential Write
0 likes · 4 min read
How Kafka Achieves High‑Performance Storage: Log Segments, Indexes, and Retention
MaGe Linux Operations
MaGe Linux Operations
Aug 19, 2025 · Big Data

Master Kafka High Availability: Replica Sync & Disaster Recovery Strategies

This article provides a comprehensive guide to building enterprise‑grade, highly available Kafka clusters, covering architecture design, hardware planning, production‑level broker configurations, ISR management, monitoring, fault‑tolerance procedures, rolling upgrades, capacity planning, and automation scripts for seamless operations.

KafkaOperationsScaling
0 likes · 16 min read
Master Kafka High Availability: Replica Sync & Disaster Recovery Strategies
StarRocks
StarRocks
Aug 19, 2025 · Big Data

How Joydata Scaled to 150 Billion Daily Events with StarRocks: A Data Architecture Journey

Facing daily data growth from millions to 150 billion records, Joydata‑U transformed its analytics platform through three architectural stages—Hadoop, Hadoop + Trino, and finally StarRocks—introducing resource isolation, Flat JSON acceleration, and Bitmap indexing to cut query latency by up to seven times and achieve sub‑2‑minute data freshness across BI, ad‑tech, game analytics, and CRM workloads.

Bitmap IndexFlat JSONFlink
0 likes · 12 min read
How Joydata Scaled to 150 Billion Daily Events with StarRocks: A Data Architecture Journey
Tencent Cloud Middleware
Tencent Cloud Middleware
Aug 1, 2025 · Backend Development

Mastering TDMQ CKafka Production: Configurations, Partitioning, and Reliable Messaging

This guide explains how to efficiently produce messages with TDMQ CKafka, covering topic creation, partition sizing, retry policies, asynchronous sending, key/value handling, batch tuning, sticky partitioning, idempotence, ACK settings, code examples, and strategies to avoid data skew and ensure high‑throughput, low‑latency messaging.

CKafkaKafkaMessage Queue
0 likes · 18 min read
Mastering TDMQ CKafka Production: Configurations, Partitioning, and Reliable Messaging
Mike Chen's Internet Architecture
Mike Chen's Internet Architecture
Jul 24, 2025 · Big Data

Boost Kafka Consumer Throughput: Multi‑Threading, Consumer Groups & Config Tuning

This guide explains why Kafka consumer throughput matters, then details practical techniques—including multi‑threaded consumption, scaling with consumer groups, client parameter tuning, and batch processing—to dramatically increase throughput while maintaining reliability in high‑concurrency, large‑scale data pipelines.

ConsumerKafkahigh-throughput
0 likes · 5 min read
Boost Kafka Consumer Throughput: Multi‑Threading, Consumer Groups & Config Tuning
Lakehouse Research Base
Lakehouse Research Base
Jul 24, 2025 · Big Data

Two Solutions for ElasticSearch Wildcard Index Collection: Flink ETL Outperforms Logstash/Kafka Pipeline

The author compares two architectures for collecting ElasticSearch wildcard index data when DataX lacks support: a Logstash-Kafka-MaxCompute pipeline with existing tools but longer chain, and a simpler Flink ETL approach writing directly to MaxCompute, concluding Flink is preferable due to lower cost and operational complexity.

DataXETLElasticsearch
0 likes · 3 min read
Two Solutions for ElasticSearch Wildcard Index Collection: Flink ETL Outperforms Logstash/Kafka Pipeline
macrozheng
macrozheng
Jul 17, 2025 · Backend Development

How We Resolved Repeated Kafka Message Backlogs in a High‑Traffic Restaurant System

This article recounts a series of Kafka message backlog incidents in a restaurant ordering system and explains how targeted optimizations—batch database queries, index tuning, data archiving, and thread‑pool scaling—eliminated the delays and restored reliable kitchen display performance.

Backend DevelopmentKafkaMessage Queue
0 likes · 12 min read
How We Resolved Repeated Kafka Message Backlogs in a High‑Traffic Restaurant System
Architect
Architect
Jul 16, 2025 · Backend Development

Mastering Spring Transaction Hooks: Async Kafka Logging After Commit

This article explains how to use Spring's TransactionSynchronizationManager to detect active transactions and reliably send Kafka messages either immediately or after transaction commit, illustrated with a payment‑system use case and complete code examples.

JavaKafkaSpring
0 likes · 10 min read
Mastering Spring Transaction Hooks: Async Kafka Logging After Commit
Mike Chen's Internet Architecture
Mike Chen's Internet Architecture
Jul 14, 2025 · Backend Development

Kafka Producer Idempotency: PID, Sequence Numbers, and Broker Deduplication

Kafka ensures that a producer’s repeated message sends, caused by network glitches or broker failures, result in only one persisted record per partition by using a unique Producer ID, monotonically increasing sequence numbers, and broker-side tracking of the latest committed sequence for each PID‑partition pair.

BackendKafkamessage deduplication
0 likes · 5 min read
Kafka Producer Idempotency: PID, Sequence Numbers, and Broker Deduplication
Mike Chen's Internet Architecture
Mike Chen's Internet Architecture
Jul 9, 2025 · Backend Development

How to Prevent Duplicate Consumption in Kafka: Practical Strategies

This article explains why Kafka’s at‑least‑once delivery can cause duplicate message processing, outlines the business risks of such duplicates, and presents four practical solutions—including idempotent design, manual offset commits, exactly‑once semantics, and dead‑letter queues—to ensure reliable consumption.

Dead Letter QueueIdempotencyKafka
0 likes · 6 min read
How to Prevent Duplicate Consumption in Kafka: Practical Strategies
Mike Chen's Internet Architecture
Mike Chen's Internet Architecture
Jul 7, 2025 · Big Data

How Kafka Achieves Billion-Message Throughput: Sequential Disk Writes, Page Cache, and Zero‑Copy

This article explains how Kafka sustains massive traffic by writing logs sequentially to disk, leveraging the operating system’s page cache for fast in‑memory writes, employing zero‑copy techniques like sendfile to avoid user‑space copying, and batching messages to reduce network overhead, thereby delivering high‑throughput, low‑latency streaming.

Kafkabatchinghigh-throughput
0 likes · 4 min read
How Kafka Achieves Billion-Message Throughput: Sequential Disk Writes, Page Cache, and Zero‑Copy
FunTester
FunTester
Jul 5, 2025 · Big Data

Master Kafka: Core Concepts and Performance Testing Strategies

This article explains Kafka’s high‑performance distributed streaming architecture, key components such as topics, partitions, producers, consumers, brokers, offsets, and ZooKeeper, and provides step‑by‑step workflows for producers and consumers along with performance‑testing tips and Maven setup.

JavaKafkaMessage Queue
0 likes · 9 min read
Master Kafka: Core Concepts and Performance Testing Strategies
Big Data Tech Team
Big Data Tech Team
Jul 3, 2025 · Big Data

Master Kafka: A Complete Learning Roadmap from Basics to Advanced Projects

This guide presents a step‑by‑step Kafka learning roadmap covering core concepts, architecture, configuration, monitoring tools, practical project ideas, advanced components like Streams and KSQL, plus code samples and resource recommendations to help beginners become proficient in real‑time data streaming.

Code ExamplesKafkaTutorial
0 likes · 14 min read
Master Kafka: A Complete Learning Roadmap from Basics to Advanced Projects
dbaplus Community
dbaplus Community
Jun 25, 2025 · Operations

How We Boosted Kafka Production Capacity by 35% with Simple Compression Tweaks

Facing petabyte‑scale log traffic, the Qunar team identified low compression rates in their Kafka‑Filebeat pipeline as the main bottleneck and, through systematic tuning of batch size, memory queues, and round‑robin settings, achieved a 35% reduction in traffic and a 30‑42% drop in request volume while raising per‑minute throughput by 35%.

BackendKafkaOperations
0 likes · 10 min read
How We Boosted Kafka Production Capacity by 35% with Simple Compression Tweaks
Code Ape Tech Column
Code Ape Tech Column
Jun 20, 2025 · Backend Development

Mastering Asynchronous Processing in Java: Build a Robust SDK with Spring

This article introduces a generic asynchronous processing SDK for Java, explaining its purpose, advantages, underlying principles, component choices such as Kafka, XXL‑Job, MySQL, and Spring AOP, and provides detailed configuration, database schema, usage examples, and best‑practice guidelines to achieve reliable, non‑blocking execution and eventual consistency in backend systems.

BackendKafkaSDK
0 likes · 9 min read
Mastering Asynchronous Processing in Java: Build a Robust SDK with Spring
Su San Talks Tech
Su San Talks Tech
Jun 20, 2025 · Backend Development

Why Environment Setup Is the Real Pain in Spring Boot Projects (And How to Fix It)

This article walks through the common frustrations of IDE and environment setup for Spring Boot projects, explains version compatibility between Spring Cloud, Spring Boot, and Kafka, demonstrates how Maven and auto‑configuration simplify dependencies, and provides ready‑to‑use code snippets and useful tooling recommendations for backend developers.

IDE setupKafkaMaven
0 likes · 11 min read
Why Environment Setup Is the Real Pain in Spring Boot Projects (And How to Fix It)
Senior Tony
Senior Tony
Jun 18, 2025 · Backend Development

Mastering High‑Concurrency Account Balance Deductions with Kafka, Sharding, and Parallel Processing

This article examines the challenges of high‑concurrency write scenarios in ad‑billing systems, explains why row‑level locking becomes a bottleneck, and presents a step‑by‑step solution using Kafka for asynchronous peak shaving, parallel consumer processing, database sharding, hotspot dispersion, and batch deduction to achieve reliable real‑time balance updates.

KafkaParallel ProcessingSharding
0 likes · 8 min read
Mastering High‑Concurrency Account Balance Deductions with Kafka, Sharding, and Parallel Processing
Architect
Architect
Jun 6, 2025 · Backend Development

How to Prevent Duplicate Messages in Kafka and Pulsar: A Practical Guide

This article explains the three message delivery semantics, the common causes of duplicate messages in queue systems, and presents concrete producer‑side, broker‑side, and consumer‑side deduplication techniques for Kafka and Pulsar, including code samples and best‑practice recommendations.

BackendKafkaMessage Queue
0 likes · 9 min read
How to Prevent Duplicate Messages in Kafka and Pulsar: A Practical Guide
Instant Consumer Technology Team
Instant Consumer Technology Team
Jun 5, 2025 · Big Data

Mastering Kafka in Production: Boost Throughput, Ensure Reliability, and Avoid Data Loss

This article shares practical Kafka production insights, covering architecture overview, producer throughput tuning, message loss prevention, broker and consumer configurations, duplicate consumption avoidance, backlog mitigation, ordering guarantees, and the mechanics of consumer group rebalancing, helping engineers build stable, high‑performance streaming pipelines.

Consumer GroupKafkaMessage Queue
0 likes · 15 min read
Mastering Kafka in Production: Boost Throughput, Ensure Reliability, and Avoid Data Loss
Mike Chen's Internet Architecture
Mike Chen's Internet Architecture
Jun 3, 2025 · Big Data

Kafka High-Concurrency Core Design Explained

This article explains how Kafka achieves high concurrency through its distributed broker cluster, partitioned topics, sequential log writes, message compression, asynchronous producer mechanisms, and OS page‑cache techniques, illustrating the combined architectural and performance optimizations that enable massive throughput.

Kafkaasynchronous producerdistributed architecture
0 likes · 4 min read
Kafka High-Concurrency Core Design Explained
Full-Stack Internet Architecture
Full-Stack Internet Architecture
May 27, 2025 · Big Data

Understanding Event Streaming in Kafka: Core Concepts, Architecture, and Use Cases

This article explains Kafka's event streaming concept, detailing events and streams, core components such as producers, topics, partitions, consumers, persistence, and typical real‑time data pipeline, event‑driven architecture, stream processing, and log aggregation use cases, highlighting its role as a foundational big‑data infrastructure.

Kafkaevent streamingmessage queues
0 likes · 7 min read
Understanding Event Streaming in Kafka: Core Concepts, Architecture, and Use Cases
Java Captain
Java Captain
May 23, 2025 · Backend Development

Common Causes of Kafka Message Loss and Mitigation Strategies

This article examines the typical reasons Kafka messages are lost across producers, brokers, and consumers, and provides detailed configuration recommendations and best‑practice solutions to significantly reduce the risk of data loss in distributed streaming systems.

ConsumerKafkaMessage Loss
0 likes · 15 min read
Common Causes of Kafka Message Loss and Mitigation Strategies