BestHub
Sign in
Tagged articles

Kafka

1415 articles · Page 8 of 15
vivo Internet Technology
vivo Internet Technology
May 18, 2022 · Backend Development

Kafka Cluster Fault Analysis: Root Cause and Cascading Failure Mechanism

A Kafka cluster at vivo suffered a total traffic drop across a resource group when a broker’s disk failed, because the default producer partitioner still hashed keys to the failed partition, exhausting client buffers and blocking all healthy partitions, prompting recommendations to avoid keys or use custom partitioners.

Fault AnalysisKafkaPerformance Optimization
0 likes · 9 min read
Kafka Cluster Fault Analysis: Root Cause and Cascading Failure Mechanism
dbaplus Community
dbaplus Community
May 17, 2022 · Backend Development

How to Size a Kafka Cluster for Over 1 Billion Daily Requests

This article walks through a scenario‑driven capacity assessment for a production‑grade Kafka cluster, covering QPS calculations, storage needs, physical machine count, disk choices, memory, CPU, network bandwidth, deployment steps, and a final resource summary.

Backend DevelopmentCluster SizingKafka
0 likes · 13 min read
How to Size a Kafka Cluster for Over 1 Billion Daily Requests
JavaEdge
JavaEdge
May 14, 2022 · Backend Development

Unveiling Kafka’s Triple‑High Architecture: Availability, Performance, and Concurrency

This article breaks down Kafka’s high‑availability, high‑performance, and high‑concurrency design, covering controller and leader election, replica and ISR mechanisms, ACK settings, the Reactor NIO model, zero‑copy I/O, compression, producer batching, memory‑pooling, and the multi‑layer network threading architecture.

KafkaNetwork ConcurrencyProducer Batching
0 likes · 21 min read
Unveiling Kafka’s Triple‑High Architecture: Availability, Performance, and Concurrency
DeWu Technology
DeWu Technology
May 9, 2022 · Backend Development

Common Issues and Solutions for Message Queue Middleware

Message‑queue middleware such as RabbitMQ, RocketMQ, ActiveMQ, and Kafka introduces challenges like ordering, loss, duplication, back‑pressure and delayed delivery, which can be mitigated by using single‑consumer queues or partitioning, enabling acknowledgments and replication, applying idempotent identifiers, scaling consumers, and employing dead‑letter or scheduling mechanisms.

KafkaMQRabbitMQ
0 likes · 21 min read
Common Issues and Solutions for Message Queue Middleware
macrozheng
macrozheng
May 5, 2022 · Backend Development

Choosing the Right Message Queue: Kafka vs RabbitMQ vs RocketMQ vs ActiveMQ Explained

An in‑depth guide compares the four major message‑queue middleware—Kafka, RabbitMQ, RocketMQ and ActiveMQ—detailing their architectures, modes, evaluation criteria, strengths, weaknesses, and selection considerations to help engineers and interviewees answer critical MQ interview questions and make informed technology choices.

ActiveMQArchitectureKafka
0 likes · 47 min read
Choosing the Right Message Queue: Kafka vs RabbitMQ vs RocketMQ vs ActiveMQ Explained
Programmer DD
Programmer DD
May 3, 2022 · Backend Development

Mastering Spring Kafka Listeners: Containers, Concurrency, and @KafkaListener

This article explains how Spring Kafka’s KafkaMessageListenerContainer and ConcurrentMessageListenerContainer work, details their start-up process and internal consumer threads, shows how @KafkaListener bridges business logic, and provides configuration examples for single‑message and batch processing in Spring Boot.

Backend DevelopmentJavaKafka
0 likes · 11 min read
Mastering Spring Kafka Listeners: Containers, Concurrency, and @KafkaListener
Code Ape Tech Column
Code Ape Tech Column
Apr 23, 2022 · Big Data

Kafka Fundamentals: Architecture, Replication, Partitioning, and Performance

This article provides a comprehensive overview of Kafka, covering its role as a message middleware, core concepts, architecture, replication management, partition handling, producer sending modes, partition assignment strategies, load balancing, reliability mechanisms, consumer models, controller election, and factors that affect its high throughput and potential message loss scenarios.

KafkaPartitioningReplication
0 likes · 29 min read
Kafka Fundamentals: Architecture, Replication, Partitioning, and Performance
Architecture Digest
Architecture Digest
Apr 18, 2022 · Operations

Zero‑Downtime Migration of 2000 Microservices to a Multi‑Cluster Managed Kafka at Wix

This article details how Wix migrated its 2000 microservices from a self‑hosted Kafka cluster to a multi‑cluster Confluent Cloud setup with zero downtime, describing the design decisions, partitioning strategy, Greyhound SDK usage, replication service, migration orchestrator, and best‑practice recommendations.

Confluent CloudGreyhound SDKKafka
0 likes · 17 min read
Zero‑Downtime Migration of 2000 Microservices to a Multi‑Cluster Managed Kafka at Wix
Tencent Cloud Developer
Tencent Cloud Developer
Apr 14, 2022 · Big Data

Understanding Kafka's Transition from ZooKeeper to KRaft: Architecture, Installation, Raft Algorithm, and Common Issues

The article explains how Kafka 3.x replaces ZooKeeper with the internal KRaft consensus layer, detailing the Raft‑based metadata storage, step‑by‑step KRaft cluster installation and configuration, and covering related concepts such as leader election, consumer‑group rebalancing, reliability settings, and performance optimizations.

InstallationKRaftKafka
0 likes · 41 min read
Understanding Kafka's Transition from ZooKeeper to KRaft: Architecture, Installation, Raft Algorithm, and Common Issues
Big Data Technology & Architecture
Big Data Technology & Architecture
Apr 11, 2022 · Big Data

Real-Time Data Warehouse Construction: Background, Objectives, Architecture, and Case Studies

This article explains the growing demand for real‑time data warehouses, outlines their objectives and layered architecture, and presents detailed case studies from Didi, Kuaishou, Tencent, Youzan and others, illustrating design choices, implementation challenges, and best practices for building scalable streaming data platforms.

ClickHouseFlinkKafka
0 likes · 48 min read
Real-Time Data Warehouse Construction: Background, Objectives, Architecture, and Case Studies
MaGe Linux Operations
MaGe Linux Operations
Apr 7, 2022 · Operations

Master ELK Stack: From Basics to Full‑Scale Log Management

This article introduces the ELK stack components, explains why centralized logging is essential, outlines core log‑system features, compares three ELK architectures, provides step‑by‑step installation and configuration for Filebeat, Logstash, Elasticsearch and Kibana, and demonstrates a complete pipeline using Kafka with code examples and diagrams.

ELKElasticsearchFilebeat
0 likes · 25 min read
Master ELK Stack: From Basics to Full‑Scale Log Management
JavaEdge
JavaEdge
Apr 6, 2022 · Big Data

When Does Kafka Lose Data? Proven Strategies to Prevent Message Loss

This article explains Kafka's message delivery semantics, identifies the exact scenarios where data can be lost in producer, broker, and consumer components, and provides concrete configuration and coding practices to ensure reliable, at‑least‑once or exactly‑once delivery in production environments.

ConsumerData ReliabilityKafka
0 likes · 18 min read
When Does Kafka Lose Data? Proven Strategies to Prevent Message Loss
21CTO
21CTO
Apr 4, 2022 · Backend Development

Choosing the Right Message Broker: RabbitMQ vs Kafka vs Redis for Microservices

This article compares three popular open‑source message brokers—RabbitMQ, Kafka, and Redis—examining their scalability, persistence, consumer models, and ideal use cases to help developers select the most suitable asynchronous communication solution for their microservice architecture.

BackendKafkaMessage Broker
0 likes · 8 min read
Choosing the Right Message Broker: RabbitMQ vs Kafka vs Redis for Microservices
Ziru Technology
Ziru Technology
Apr 1, 2022 · Fundamentals

How Kafka’s Hierarchical Timing Wheel Optimizes Task Scheduling

This article explains the time‑wheel algorithm, from its basic circular‑buffer principle to simple, round‑based, and hierarchical variants, and shows how Kafka implements a multi‑level timing wheel to achieve efficient, low‑memory delayed task execution.

Kafkabackend algorithmshierarchical timer
0 likes · 13 min read
How Kafka’s Hierarchical Timing Wheel Optimizes Task Scheduling
IT Architects Alliance
IT Architects Alliance
Mar 30, 2022 · Backend Development

Designing a Scalable Event-Driven Microservices Architecture for E‑Commerce

This article guides readers through the evolution from a monolithic e‑commerce application to a highly available, scalable, low‑latency event‑driven microservices architecture, covering design patterns, principles, API gateway, BFF, service aggregation, CQRS, event sourcing, and technology stack choices such as Kafka and Cassandra.

ArchitectureCQRSKafka
0 likes · 21 min read
Designing a Scalable Event-Driven Microservices Architecture for E‑Commerce
dbaplus Community
dbaplus Community
Mar 27, 2022 · Backend Development

When to Choose Kafka Over RabbitMQ: A Six‑Scenario Comparison

This article compares Kafka and RabbitMQ across six real‑world scenarios—message ordering, routing, timeout handling, persistence, error handling, and throughput—to help engineers decide which message queue best fits their system requirements.

KafkaRabbitMQSystem Design
0 likes · 12 min read
When to Choose Kafka Over RabbitMQ: A Six‑Scenario Comparison
Su San Talks Tech
Su San Talks Tech
Mar 27, 2022 · Big Data

Top 10 Advanced Kafka Interview Questions with In‑Depth Answers

This article provides a comprehensive collection of advanced Kafka interview questions covering core architecture, storage mechanisms, replication, leader election, controller responsibilities, consumer group rebalancing, message semantics, partition assignment strategies, performance tuning, and practical tips for handling large message backlogs.

ConsumerGroupKafkaReplication
0 likes · 33 min read
Top 10 Advanced Kafka Interview Questions with In‑Depth Answers
StarRocks
StarRocks
Mar 23, 2022 · Databases

Accelerating Zepp Health’s Analytics with StarRocks: An OLAP Case Study

Facing inflexible point‑lookup limits and slow query times on HBase, Zepp Health redesigned its massive event‑tracking data pipeline—migrating ingestion through Kafka, Flink, and Hudi to a StarRocks‑based OLAP layer—achieving sub‑100 ms average query latency, 20 % storage savings, and dramatically faster multi‑dimensional analytics.

FlinkHudiKafka
0 likes · 9 min read
Accelerating Zepp Health’s Analytics with StarRocks: An OLAP Case Study
dbaplus Community
dbaplus Community
Mar 22, 2022 · Operations

How Meituan Scales Kafka to 7,500 Nodes: Real-World Optimizations and Lessons

Meituan’s data platform runs Kafka on over 7,500 machines, handling daily traffic exceeding 21 PB, and tackles latency, slow nodes, and massive cluster management through layered optimizations—including disk balancing, pipeline acceleration, fetcher isolation, cgroup isolation, SSD caching, isolation strategies, full‑link monitoring, lifecycle management, and TOR disaster recovery.

KafkaLarge-scale systemsdata platform
0 likes · 23 min read
How Meituan Scales Kafka to 7,500 Nodes: Real-World Optimizations and Lessons
Top Architect
Top Architect
Mar 15, 2022 · Backend Development

Understanding Apache Kafka Replication Mechanism and Its Design Principles

This article explains Apache Kafka's replication mechanism, covering the benefits of data redundancy, the roles of leader and follower replicas, the In‑Sync Replica (ISR) concept, unclean leader election, and how these design choices affect availability, consistency, and scalability in distributed systems.

ISRKafkaReplication
0 likes · 11 min read
Understanding Apache Kafka Replication Mechanism and Its Design Principles
IT Architects Alliance
IT Architects Alliance
Mar 14, 2022 · Big Data

Comprehensive Guide to Kafka Architecture, Core Concepts, and Production Deployment

This article provides an in‑depth overview of Kafka, covering why messaging systems are needed, core concepts, cluster architecture, performance optimizations such as sequential disk writes and zero‑copy, hardware sizing, replication, consumer groups, offset management, rebalance strategies, and practical deployment and operational guidelines.

Cluster DeploymentDistributed MessagingKafka
0 likes · 35 min read
Comprehensive Guide to Kafka Architecture, Core Concepts, and Production Deployment
Alibaba Cloud Developer
Alibaba Cloud Developer
Mar 7, 2022 · Big Data

How China Mobile’s Real‑Time Computing Platform Scales Billions of Events with Flink

This article details China Mobile (Suzhou) Software Technology's evolution from Storm to Flink for real‑time computing, its multi‑version engine and log‑retrieval designs, signal‑business data pipeline optimizations, stability practices around ZooKeeper, and future directions in resource scaling and data‑lake integration.

FlinkKafkacloud
0 likes · 12 min read
How China Mobile’s Real‑Time Computing Platform Scales Billions of Events with Flink
dbaplus Community
dbaplus Community
Mar 2, 2022 · Big Data

How Real‑Time Data Warehouses Power Modern Business: Architecture, Cases, and Best Practices

This article explores the growing demand for real‑time data warehouses, compares them with traditional offline warehouses, and presents detailed architectures, layer designs, naming conventions, and case studies from companies like Didi, Kuaishou, Tencent, and Youzan, highlighting challenges, solutions, and performance optimizations.

Big Data ArchitectureFlinkKafka
0 likes · 47 min read
How Real‑Time Data Warehouses Power Modern Business: Architecture, Cases, and Best Practices
Architect's Journey
Architect's Journey
Mar 2, 2022 · Backend Development

Interview Basics: How to Guarantee Message Reliability in MQ

The article explains how to achieve 100% message delivery and consumption in MQ systems by covering producer acknowledgments, broker persistence mechanisms, and consumer idempotency, with detailed comparisons of RabbitMQ and Kafka implementations and configuration tips.

ACKKafkaPersistence
0 likes · 13 min read
Interview Basics: How to Guarantee Message Reliability in MQ
Efficient Ops
Efficient Ops
Feb 23, 2022 · Operations

Why a Single Kafka Broker Crash Can Halt All Consumers – The HA Explained

An in‑depth look at Kafka’s high‑availability architecture reveals how multi‑replica redundancy, ISR mechanisms, and the configuration of the __consumer_offset topic interact, explaining why a single broker failure can render the entire cluster unusable and how to properly configure replication and ack settings to prevent it.

ACKConsumer offsetISR
0 likes · 10 min read
Why a Single Kafka Broker Crash Can Halt All Consumers – The HA Explained
Top Architect
Top Architect
Feb 20, 2022 · Big Data

Understanding Kafka’s Core Design: Topics, Partitions, Replicas, Consumer Groups, and Performance Optimizations

This article explains Kafka’s fundamental architecture—including topics, partitions, replication, consumer groups, cluster coordination, and performance techniques such as sequential writes, zero‑copy, and log segmentation—to help readers improve their design and coding skills for large‑scale messaging systems.

KafkaPerformance Optimization
0 likes · 12 min read
Understanding Kafka’s Core Design: Topics, Partitions, Replicas, Consumer Groups, and Performance Optimizations
Tencent Cloud Developer
Tencent Cloud Developer
Feb 15, 2022 · Backend Development

Message Queue Middleware Selection Guide: Kafka vs Pulsar vs RocketMQ vs RabbitMQ vs NSQ

This guide compares Kafka, Pulsar, RocketMQ, RabbitMQ, and NSQ across architecture, consumption models, delay and dead‑letter support, persistence, ordering, multi‑tenant isolation, and performance, recommending Kafka for high‑throughput log processing, Pulsar for transactional delayed messaging and disaster recovery, and NSQ for Go‑centric custom solutions.

KafkaMessage StreamingNSQ
0 likes · 26 min read
Message Queue Middleware Selection Guide: Kafka vs Pulsar vs RocketMQ vs RabbitMQ vs NSQ
Selected Java Interview Questions
Selected Java Interview Questions
Feb 5, 2022 · Backend Development

Message Queue Fundamentals: Use Cases, Product Comparison, High Availability, and Reliability Strategies

This article explains why message queues are used, outlines common scenarios such as decoupling, asynchronous processing and traffic shaping, compares major MQ products, and provides practical guidance on high availability, preventing loss, duplicate consumption, ordering, backlog handling, and expiration.

KafkaRabbitMQRocketMQ
0 likes · 8 min read
Message Queue Fundamentals: Use Cases, Product Comparison, High Availability, and Reliability Strategies
IT Architects Alliance
IT Architects Alliance
Feb 3, 2022 · Backend Development

Common Issues in Message Queues and Distributed Transaction Solutions

This article explains the typical problems encountered with message queues, such as message loss, duplicate delivery, and distributed transaction handling, and details various solutions including local message tables, MQ‑based transactions, and the specific mechanisms used by RocketMQ, Kafka, and RabbitMQ to ensure reliability and consistency.

KafkaMQRabbitMQ
0 likes · 20 min read
Common Issues in Message Queues and Distributed Transaction Solutions
DataFunTalk
DataFunTalk
Feb 1, 2022 · Big Data

Kafka at Meituan: Practices, Challenges, and Optimizations for Large‑Scale Data Platforms

This article presents Meituan's large‑scale Kafka deployment, describing the current state and challenges of massive data ingestion, detailing latency‑reduction techniques, cluster‑level optimizations, SSD‑based caching, isolation strategies, full‑link monitoring, lifecycle management, and future directions for high availability.

KafkaMeituanRead‑Write Latency
0 likes · 22 min read
Kafka at Meituan: Practices, Challenges, and Optimizations for Large‑Scale Data Platforms
Top Architect
Top Architect
Jan 31, 2022 · Backend Development

Delayed Queue Technology Research and Implementation Overview

This article surveys various delayed queue implementations—including Kafka, RocketMQ, Redis (Redisson), and Netty’s HashedWheelTimer—examining their design principles, advantages, drawbacks, and integration strategies, and proposes a unified micro‑service architecture leveraging Kafka topics, Redis ZSETs, and thread‑pool optimizations for reliable message scheduling.

KafkaMicroservicesNetty
0 likes · 15 min read
Delayed Queue Technology Research and Implementation Overview
Aikesheng Open Source Community
Aikesheng Open Source Community
Jan 27, 2022 · Databases

DTLE 4.22.01.0 Release Notes – New Features and Bug Fixes

DTLE 4.22.01.0, an open‑source MySQL‑focused data transfer component, has been released, introducing Oracle‑MySQL incremental DDL/DML sync, enhanced WHERE clause support, initial Kafka schema‑change handling, and numerous bug fixes including binlog relay issues, character‑set handling, and CREATE TABLE … LIKE renaming problems.

DTLEData TransferKafka
0 likes · 4 min read
DTLE 4.22.01.0 Release Notes – New Features and Bug Fixes
HomeTech
HomeTech
Jan 26, 2022 · Operations

Design and Practice of Autohome's Performance Testing Platform PTS

The article details the architecture, key components, testing types, and operational results of Autohome's PTS platform, which uses Docker Swarm, gRPC, JMeter, Flume‑Kafka, and Flink to conduct large‑scale distributed load testing for the 818 event and outlines future improvements toward Kubernetes and direct Kafka logging.

Docker SwarmFlinkJMeter
0 likes · 8 min read
Design and Practice of Autohome's Performance Testing Platform PTS
ByteDance Data Platform
ByteDance Data Platform
Jan 17, 2022 · Big Data

How ByteHouse Scales Real‑Time Analytics on ClickHouse: Challenges & Solutions

This article details ByteHouse’s evolution from ClickHouse, presenting two real‑time analytics use cases, the technical selection process, performance bottlenecks such as write throughput and Kafka consumption, and the engineered solutions—including asynchronous indexing, multi‑threaded Kafka engines, and enhanced Buffer engines—that enable reliable, high‑throughput data processing at massive scale.

ByteHouseClickHouseKafka
0 likes · 11 min read
How ByteHouse Scales Real‑Time Analytics on ClickHouse: Challenges & Solutions
Big Data Technology & Architecture
Big Data Technology & Architecture
Jan 12, 2022 · Big Data

Common Production Issues and Troubleshooting Guide for Apache Flink

This article compiles a comprehensive list of common production problems encountered with Apache Flink, covering cluster sizing, checkpoint failures, backpressure analysis, resource allocation, deployment errors, UDF definitions, data skew, Kafka configurations, and provides detailed troubleshooting steps and best‑practice recommendations.

Apache FlinkCheckpointKafka
0 likes · 39 min read
Common Production Issues and Troubleshooting Guide for Apache Flink
Java Interview Crash Guide
Java Interview Crash Guide
Jan 12, 2022 · Backend Development

Designing Scalable Delayed Queues: Kafka, RocketMQ, Redis & More

This article surveys delayed‑queue implementations, comparing Kafka, RocketMQ, and Redis (Redisson) designs, outlining their architectures, pros and cons, integration details, timing‑wheel mechanisms, and practical considerations for building a reliable distributed delayed‑queue service.

Kafkadelayed queuedistributed systems
0 likes · 13 min read
Designing Scalable Delayed Queues: Kafka, RocketMQ, Redis & More
Su San Talks Tech
Su San Talks Tech
Jan 7, 2022 · Big Data

Unlock Kafka’s Speed: Deep Dive into Performance Optimizations

This article explores Kafka’s performance architecture, covering network and disk bottlenecks, sequential writes, zero‑copy techniques, page cache usage, Reactor‑based networking, batch processing, compression, partition concurrency, and file structures, providing practical optimization methods for high‑throughput streaming applications.

JavaKafkaOptimization
0 likes · 17 min read
Unlock Kafka’s Speed: Deep Dive into Performance Optimizations
21CTO
21CTO
Dec 25, 2021 · Backend Development

Message Queue Mastery: Interview Q&A, Pros/Cons, and Choosing the Best MQ

This article explains why message queues are used, outlines core interview questions, analyzes interviewers' expectations, compares major MQ products, and provides practical guidance on selecting the right queue for different scenarios.

KafkaRabbitMQbackend architecture
0 likes · 12 min read
Message Queue Mastery: Interview Q&A, Pros/Cons, and Choosing the Best MQ
21CTO
21CTO
Dec 23, 2021 · Backend Development

Choosing the Right Message Broker for Microservices: RabbitMQ vs Kafka vs Redis

This article explains the difference between synchronous and asynchronous microservice communication, outlines the benefits of async messaging, and compares three popular message brokers—RabbitMQ, Kafka, and Redis—based on scale, persistence, and consumer patterns, offering guidance on selecting the right broker for various use cases.

KafkaRabbitMQRedis
0 likes · 8 min read
Choosing the Right Message Broker for Microservices: RabbitMQ vs Kafka vs Redis
Architecture Digest
Architecture Digest
Dec 20, 2021 · Backend Development

Understanding Kafka: Core Design, Architecture, and Performance

This article explains Kafka’s fundamental design concepts—including topics, partitions, replicas, consumer groups, and its network architecture—while highlighting performance features such as sequential writes, zero‑copy, log segmentation, and how the controller coordinates with ZooKeeper, providing a comprehensive overview for backend developers.

Backend DevelopmentKafkabig data
0 likes · 12 min read
Understanding Kafka: Core Design, Architecture, and Performance
HelloTech
HelloTech
Dec 13, 2021 · Big Data

Smart Matching Engine for Ride-Sharing: Technical Implementation and Algorithms

The Smart Matching Engine for Haolo’s ride‑sharing service ingests driver and passenger orders via Kafka‑Flink pipelines into Elasticsearch, then applies multi‑stage matching—nearby search, itinerary‑based filtering using ETA, angle, distance, route‑similarity and shared‑mileage calculations—and finally ranks results with evolving pre‑sorting and algorithmic models, including PMML and deep‑learning, to optimize driver‑passenger pairing.

ElasticsearchFlinkKafka
0 likes · 9 min read
Smart Matching Engine for Ride-Sharing: Technical Implementation and Algorithms
Wukong Talks Architecture
Wukong Talks Architecture
Dec 8, 2021 · Big Data

Understanding Kafka Core Concepts: Architecture, Messaging Models, Partitioning, Consumer Groups, and Reliability

This article provides a comprehensive overview of Kafka, covering its layered architecture with Zookeeper, core concepts such as topics, partitions and consumer groups, communication workflow, partition selection strategies, rebalancing mechanisms, reliability configurations, replica synchronization, and reasons for moving away from Zookeeper, all explained in clear English.

Kafkaconsumer groupsdistributed systems
0 likes · 19 min read
Understanding Kafka Core Concepts: Architecture, Messaging Models, Partitioning, Consumer Groups, and Reliability
DataFunSummit
DataFunSummit
Dec 6, 2021 · Big Data

Design and Performance Optimization of a Real‑Time Billion‑Scale Data Processing Pipeline

This article reviews the background, architecture, and a series of performance‑optimizing techniques—including consumption, batch, storage, and execution‑engine tweaks—applied to a real‑time pipeline that processes hundreds of billions of records daily, and presents the resulting resource savings and latency improvements.

KafkaPerformance OptimizationSparkSQL
0 likes · 9 min read
Design and Performance Optimization of a Real‑Time Billion‑Scale Data Processing Pipeline
IT Architects Alliance
IT Architects Alliance
Dec 3, 2021 · Backend Development

Design and Performance Optimization of a High‑Concurrency WebSocket Gateway (Version 2.0)

This article details the evolution from a Node.js‑based WebSocket gateway to a Go‑implemented, gRPC‑driven architecture, describing the redesign of connection handling, TLS off‑loading, socket ID generation, session management, custom Kafka headers, code‑level optimizations, and extensive performance testing that validates the new gateway’s scalability and resource efficiency.

GoKafkaPerformance
0 likes · 22 min read
Design and Performance Optimization of a High‑Concurrency WebSocket Gateway (Version 2.0)
Top Architect
Top Architect
Dec 2, 2021 · Backend Development

Design and Performance Optimization of a High‑Scale WebSocket Gateway (Version 2.0)

This article describes the architectural evolution from Gateway 1.0 to Gateway 2.0 for a high‑traffic document collaboration platform, detailing the redesign of the WebSocket layer, resource‑usage optimizations, heartbeat mechanisms, custom Kafka headers, message serialization, load‑testing results, and the operational lessons learned to support millions of concurrent connections.

GoKafkaMicroservices
0 likes · 22 min read
Design and Performance Optimization of a High‑Scale WebSocket Gateway (Version 2.0)
Architecture Digest
Architecture Digest
Dec 2, 2021 · Backend Development

Design and Performance Optimization of a Million‑Scale WebSocket Gateway at Shimo Docs

This article details the redesign of Shimo Docs' WebSocket gateway—from a Node.js/Socket.IO 1.0 version to a Go‑based 2.0 architecture—covering handshake degradation, TLS memory savings, SnowFlake SocketID generation, Redis‑based session broadcasting, heartbeat tuning, custom Kafka headers, object pooling, MessagePack compression, extensive performance testing, and the resulting stability and scalability improvements for handling half‑a‑million concurrent connections.

GoKafkaMicroservices
0 likes · 23 min read
Design and Performance Optimization of a Million‑Scale WebSocket Gateway at Shimo Docs
Hacker Afternoon Tea
Hacker Afternoon Tea
Dec 1, 2021 · Big Data

Setting Up a Local Snuba Development Environment for Sentry Monitoring

This guide walks through cloning the Sentry and Snuba repositories, installing macOS system dependencies, configuring Python and Rust toolchains, launching Docker containers for ClickHouse, Kafka and Redis, running Snuba migrations, and finally starting the dev server to query events via a local UI.

ClickHouseDevOpsDocker
0 likes · 9 min read
Setting Up a Local Snuba Development Environment for Sentry Monitoring
21CTO
21CTO
Nov 30, 2021 · Backend Development

Building a Million-Scale WebSocket Gateway: Architecture, Optimization & Performance

This article details the design, refactoring, and performance testing of a high‑traffic WebSocket gateway for Shimo Docs, covering the evolution from a Node.js Socket.IO version to a Go‑based microservice architecture, TLS memory tuning, socket ID generation, heartbeat handling, custom Kafka headers, and resource‑efficient scaling to half‑a‑million concurrent connections.

GoKafkaRedis
0 likes · 25 min read
Building a Million-Scale WebSocket Gateway: Architecture, Optimization & Performance
High Availability Architecture
High Availability Architecture
Nov 30, 2021 · Backend Development

Design and Performance Optimization of a Million‑Scale WebSocket Gateway

This article details the evolution from a Node.js Socket.IO gateway to a Go‑based, gRPC‑enabled WebSocket gateway that supports millions of concurrent connections, covering architecture redesign, TLS handling, socket ID generation, session management, heartbeat optimization, custom Kafka headers, code‑level refinements, and extensive performance testing results.

GoKafkaPerformance
0 likes · 21 min read
Design and Performance Optimization of a Million‑Scale WebSocket Gateway
dbaplus Community
dbaplus Community
Nov 22, 2021 · Databases

Transforming MySQL Monitoring: From Nagios to Kafka‑Powered Alerts

Qunar’s DBA team overhauled their MySQL monitoring and alert system—originally built on Nagios and NRPE—by integrating a Kafka‑based pipeline, a custom alarm service, and MySQL‑stored alert templates, achieving flexible thresholds, granular silencing, high‑availability processing, and early‑stage intelligent management of alerts, slow queries, and disk space.

AlertingDBAKafka
0 likes · 14 min read
Transforming MySQL Monitoring: From Nagios to Kafka‑Powered Alerts
Big Data Technology & Architecture
Big Data Technology & Architecture
Nov 22, 2021 · Big Data

Comprehensive Big Data Learning Path and Resource Guide

This article presents a detailed learning roadmap for aspiring big‑data experts, covering foundational programming languages, data structures, Linux basics, databases, distributed system theory, and essential frameworks such as Hadoop, Spark, Flink, Kafka, and provides curated B‑site video links and reference materials.

FlinkHadoopKafka
0 likes · 9 min read
Comprehensive Big Data Learning Path and Resource Guide
IT Architects Alliance
IT Architects Alliance
Nov 22, 2021 · Backend Development

Understanding Message Queues, JMS, and Kafka: A Practical Guide

This article demystifies message queues by comparing them to parcel lockers, explains their role in asynchronous processing and system decoupling, details point‑to‑point and publish‑subscribe models, introduces JMS, AMQP, RabbitMQ, ActiveMQ, RocketMQ, and Kafka, and outlines their architectures and high‑availability mechanisms.

KafkaRabbitMQasynchronous-processing
0 likes · 16 min read
Understanding Message Queues, JMS, and Kafka: A Practical Guide
Top Architect
Top Architect
Nov 19, 2021 · Big Data

Understanding Kafka’s Design: Topics, Partitions, Consumer Groups, and Cluster Architecture

This article explains Kafka’s core design concepts—including the role of a message system, topics, partitions, producers, consumers, consumer groups, replica management, controller coordination with Zookeeper, performance optimizations like sequential writes and zero‑copy, and its network thread model—illustrated with diagrams and code snippets.

Kafkacluster architectureconsumer groups
0 likes · 12 min read
Understanding Kafka’s Design: Topics, Partitions, Consumer Groups, and Cluster Architecture
Big Data Technology & Architecture
Big Data Technology & Architecture
Nov 15, 2021 · Operations

A Comprehensive Overview of Kafka Monitoring Tools

This article provides a comprehensive overview of popular Kafka monitoring solutions—including JMX, Kafka Manager (CMAK), Kafka Eagle, and Logi‑KafkaManager—detailing their features, installation steps, configuration examples, and comparative advantages, while also mentioning custom setups using JMXTrans, InfluxDB, and Grafana.

CMAKKafkaKafka Eagle
0 likes · 8 min read
A Comprehensive Overview of Kafka Monitoring Tools
IT Architects Alliance
IT Architects Alliance
Nov 15, 2021 · Industry Insights

Why Kafka’s Topic‑Partition Design Powers Scalable Messaging

This article explains Kafka’s core architecture—including topics, partitions, replication, consumer groups, controller coordination with Zookeeper, and performance tricks like sequential writes and zero‑copy—to show how it achieves high‑throughput, fault‑tolerant messaging for large‑scale systems.

Consumer GroupKafkaPerformance
0 likes · 11 min read
Why Kafka’s Topic‑Partition Design Powers Scalable Messaging
Top Architect
Top Architect
Nov 14, 2021 · Big Data

Kafka Core Concepts Illustrated: Basics, Producers/Consumers, Topics, Partitions, and Architecture

This article provides a visual walkthrough of Kafka's fundamental concepts—including its basic architecture, producer‑consumer model, topics, partitions, and cluster design—offering clear explanations and diagrams to help readers grasp how Kafka enables reliable, scalable message streaming in modern systems.

ConsumerKafkaMessage Streaming
0 likes · 5 min read
Kafka Core Concepts Illustrated: Basics, Producers/Consumers, Topics, Partitions, and Architecture
dbaplus Community
dbaplus Community
Nov 10, 2021 · Databases

How HuoLaLa Built a Hybrid‑Cloud Database Governance Platform

This article outlines HuoLaLa's journey from a fragmented multi‑cloud environment to a unified, platform‑driven database governance system, detailing the technical challenges, architectural decisions, key components for MySQL, Redis, Kafka, and other middleware, and the measurable stability and cost improvements achieved after a year of operation.

Database GovernanceKafkaMySQL
0 likes · 44 min read
How HuoLaLa Built a Hybrid‑Cloud Database Governance Platform
IT Architects Alliance
IT Architects Alliance
Nov 8, 2021 · Big Data

Kafka Core Concepts, Architecture, Performance Optimization, and Operational Practices

This comprehensive guide explains Kafka's core value for decoupling and asynchronous processing, details its producer‑consumer model, cluster architecture, log segmentation, zero‑copy I/O, hardware sizing, network planning, operational commands, throughput tuning, exception handling, consumer group mechanics, offset management, rebalance strategies, and internal mechanisms such as LEO/HW, controllers, delayed tasks, and time‑wheel scheduling.

Cluster DeploymentJavaKafka
0 likes · 31 min read
Kafka Core Concepts, Architecture, Performance Optimization, and Operational Practices
Tencent Qidian Tech Team
Tencent Qidian Tech Team
Nov 1, 2021 · Backend Development

How to Build a Scalable Distributed Timer with Redis and Time Wheel

This article explains the design of a distributed timer service using a time‑wheel data structure stored in Redis, covering application scenarios, required features, architecture components such as access layer, scheduler, worker, and management center, and detailing reliability and performance techniques.

Kafkadistributed timerhigh availability
0 likes · 13 min read
How to Build a Scalable Distributed Timer with Redis and Time Wheel
Top Architect
Top Architect
Oct 30, 2021 · Backend Development

RabbitMQ vs Kafka: Architectural Comparison and Throughput Considerations

This article compares RabbitMQ and Kafka by explaining their architectures, queue consumption and production mechanisms, highlighting the master‑mirror design of RabbitMQ versus Kafka's partitioned approach, and summarizing their throughput characteristics to help engineers choose the appropriate messaging middleware for their needs.

Backend DevelopmentKafkaRabbitMQ
0 likes · 7 min read
RabbitMQ vs Kafka: Architectural Comparison and Throughput Considerations
MaGe Linux Operations
MaGe Linux Operations
Oct 28, 2021 · Fundamentals

Demystifying Message Queues, JMS, and Kafka: A Beginner’s Guide

This article explains the fundamentals of message queues, their role in asynchronous processing and system decoupling, compares point‑to‑point and publish‑subscribe patterns, introduces Java Message Service (JMS) and its API, and outlines popular implementations such as ActiveMQ, RabbitMQ, RocketMQ, and Kafka with their architectures and high‑availability mechanisms.

AMQPKafkaRabbitMQ
0 likes · 15 min read
Demystifying Message Queues, JMS, and Kafka: A Beginner’s Guide
High Availability Architecture
High Availability Architecture
Oct 28, 2021 · Cloud Native

Design and Implementation Differences between Pulsar, Kafka, RocketMQ, and InLong‑TubeMQ

This article compares Pulsar with Kafka, RocketMQ and InLong‑TubeMQ, focusing on cloud‑native multi‑tenant design, multi‑level flow control, compute‑storage separation, partitioning strategies, metadata storage, and message delivery mechanisms, highlighting each system's architectural advantages and trade‑offs.

Cloud NativeInLong‑TubeMQKafka
0 likes · 18 min read
Design and Implementation Differences between Pulsar, Kafka, RocketMQ, and InLong‑TubeMQ
Tencent Cloud Developer
Tencent Cloud Developer
Oct 26, 2021 · Cloud Native

Design and Implementation Differences between Pulsar, Kafka, RocketMQ, and InLong‑TubeMQ

Pulsar’s cloud‑native, multi‑tenant architecture with hierarchical flow control, stateless brokers separated from BookKeeper storage, global partitions supporting multiple consumers, push‑based dispatch, and precise acknowledgment outperforms Kafka, RocketMQ, and InLong‑TubeMQ, though its reliance on ZooKeeper and higher resource usage demand careful operational planning.

Flow ControlInLong‑TubeMQKafka
0 likes · 17 min read
Design and Implementation Differences between Pulsar, Kafka, RocketMQ, and InLong‑TubeMQ
MaGe Linux Operations
MaGe Linux Operations
Oct 16, 2021 · Operations

Why Does One Kafka Broker Failure Halt All Consumers? HA & Replication Explained

The article examines Kafka’s high‑availability mechanisms, detailing its multi‑replica design, ISR synchronization, leader election, and the critical role of the __consumer_offset topic, and explains why a single broker outage can render the entire cluster unusable unless replication factors are properly configured.

Consumer offsetKafkaReplication
0 likes · 10 min read
Why Does One Kafka Broker Failure Halt All Consumers? HA & Replication Explained
Java High-Performance Architecture
Java High-Performance Architecture
Oct 14, 2021 · Operations

Build a Real‑Time Log Collection Pipeline with SpringBoot, Kafka, Filebeat, Logstash & Kibana

This guide walks through setting up a complete log‑collection and visualization pipeline—including SpringBoot log4j2 configuration, Kafka broker creation, Filebeat forwarding, Logstash processing, and Kibana dashboard setup—so you can capture, ship, and analyze application logs in real time.

ElasticsearchFilebeatKafka
0 likes · 17 min read
Build a Real‑Time Log Collection Pipeline with SpringBoot, Kafka, Filebeat, Logstash & Kibana
Architecture Digest
Architecture Digest
Oct 13, 2021 · Fundamentals

Introduction to Message Queues, JMS, and Kafka

This article explains the fundamentals of message queues, compares JMS and various MQ implementations such as RabbitMQ and Kafka, describes their architectures, usage scenarios, and key concepts like producers, consumers, topics, partitions, and high‑availability mechanisms.

AMQPKafkaRabbitMQ
0 likes · 15 min read
Introduction to Message Queues, JMS, and Kafka
Big Data Technology Architecture
Big Data Technology Architecture
Oct 9, 2021 · Big Data

Apache Kafka 3.0 Release Highlights and New Features

Apache Kafka 3.0 introduces major enhancements including KRaft consensus, deprecation of Java 8 and Scala 2.12 support, stronger producer guarantees, updated APIs, improved Kafka Connect, MirrorMaker 2 flexibility, and numerous KIP-driven feature upgrades, marking a significant step forward for the distributed streaming platform.

KafkaKafka 3.0Streaming
0 likes · 13 min read
Apache Kafka 3.0 Release Highlights and New Features
Selected Java Interview Questions
Selected Java Interview Questions
Oct 9, 2021 · Backend Development

RocketMQ vs Kafka: Detailed Feature, Performance, and Reliability Comparison

This article provides a comprehensive comparison between RocketMQ and Kafka, covering data reliability, performance, queue capacity, real‑time delivery, retry mechanisms, ordering guarantees, scheduled messages, transactional support, query capabilities, message tracing, consumer parallelism, filtering, and commercial backing, helping engineers choose the right messaging middleware for their workloads.

KafkaPerformanceRocketMQ
0 likes · 11 min read
RocketMQ vs Kafka: Detailed Feature, Performance, and Reliability Comparison
Hacker Afternoon Tea
Hacker Afternoon Tea
Oct 8, 2021 · Big Data

Understanding Snuba: Kafka‑Powered ClickHouse Data Platform for Sentry

Snuba is a ClickHouse‑based service that offers rich data models, fast Kafka ingestion, and a rule‑based query optimizer for Sentry, replacing the original Postgres‑Redis stack and supporting multiple time‑series datasets, streaming and point‑in‑time queries with configurable consistency guarantees.

ClickHouseData IngestionKafka
0 likes · 11 min read
Understanding Snuba: Kafka‑Powered ClickHouse Data Platform for Sentry
Efficient Ops
Efficient Ops
Oct 8, 2021 · Operations

Why a Single Kafka Broker Failure Can Halt the Entire Cluster

This article explains Kafka's high‑availability architecture, covering multi‑replica redundancy, ISR synchronization, producer ACK settings, and the critical role of the __consumer_offset topic, and shows how to configure replication factors to prevent a single‑node outage from stopping consumption.

Consumer offsetKafkaReplication
0 likes · 11 min read
Why a Single Kafka Broker Failure Can Halt the Entire Cluster
21CTO
21CTO
Oct 6, 2021 · Big Data

Building a Real-Time TB-Scale Bill Query System with Kafka, Kudu, and Presto

This article details the design and implementation of a real‑time, TB‑scale bill‑detail query platform that leverages Kafka for streaming, Debezium and Confluent Platform for change capture, Kudu for low‑latency storage, and Presto/Kylin for fast OLAP queries, while outlining deployment, integration, and future enhancements.

KafkaKuduReal-time Data
0 likes · 19 min read
Building a Real-Time TB-Scale Bill Query System with Kafka, Kudu, and Presto