BestHub
Sign in
Tagged articles

Kafka

1415 articles · Page 11 of 15
Big Data Technology & Architecture
Big Data Technology & Architecture
Sep 18, 2020 · Big Data

Understanding Kafka Consumer Groups, Partition Assignment, and Offset Management

This article explains how Kafka consumer groups accelerate message consumption by distributing partitions across multiple consumers, details the three key characteristics of consumer groups, and provides in‑depth guidance on partition assignment strategies and offset management with practical Java code examples.

Consumer GroupKafkaPartition Assignment
0 likes · 13 min read
Understanding Kafka Consumer Groups, Partition Assignment, and Offset Management
Architect
Architect
Sep 17, 2020 · Big Data

Kafka Exactly-Once Semantics and Transaction API Overview

This article explains Kafka's exactly‑once semantics and transaction support, detailing the new producer API methods, related exceptions, configuration parameters, and a sample application illustrating how to initialize, begin, process, and commit or abort transactions while ensuring idempotent and atomic message handling.

ConfigurationExactly-OnceIdempotence
0 likes · 19 min read
Kafka Exactly-Once Semantics and Transaction API Overview
IT Architects Alliance
IT Architects Alliance
Sep 15, 2020 · Backend Development

Step‑by‑Step Guide to Deploying a Multi‑Node Kafka Cluster

This tutorial walks through setting up a four‑node Kafka cluster—including Zookeeper installation, broker configuration, service startup, replication settings, fault handling, and leader election—using Linux commands and detailed code snippets to help readers build a production‑ready streaming platform.

BackendCluster DeploymentKafka
0 likes · 14 min read
Step‑by‑Step Guide to Deploying a Multi‑Node Kafka Cluster
JavaEdge
JavaEdge
Sep 15, 2020 · Backend Development

How Kafka Uses ZooKeeper for Metadata Management and Client Coordination

This article explains how Kafka relies on ZooKeeper to store cluster metadata, detailing the ZK node hierarchy, the process by which clients locate brokers, the broker‑side handling of metadata requests, and recommended practices for large‑scale deployments.

Backend DevelopmentKafkametadata
0 likes · 8 min read
How Kafka Uses ZooKeeper for Metadata Management and Client Coordination
DataFunTalk
DataFunTalk
Sep 13, 2020 · Big Data

Online Sample Generation with Flink: Architecture and Implementation

This article explains why Flink is chosen for online sample generation, describes the end‑to‑end implementation steps—including stream union, state‑timer processing, and output formatting—covers state backend choices, monitoring, validation, fault handling, and platformization for scalable real‑time machine‑learning pipelines.

FlinkKafkaOnline Sample Generation
0 likes · 11 min read
Online Sample Generation with Flink: Architecture and Implementation
dbaplus Community
dbaplus Community
Sep 1, 2020 · Big Data

Mastering Real‑Time MySQL Binlog Sync with Debezium, Kafka & Hive

This article presents a systematic guide to real‑time MySQL binlog ingestion, outlining three core principles—decoupling from business data, handling schema changes, and ensuring traceability—followed by concrete Debezium‑Kafka‑Hive solutions, scenario‑specific tactics, and practical tips for reliable data pipelines.

Data IngestionDebeziumHive
0 likes · 15 min read
Mastering Real‑Time MySQL Binlog Sync with Debezium, Kafka & Hive
DataFunTalk
DataFunTalk
Sep 1, 2020 · Big Data

NetEase Real-Time Computing Platform (Sloth): Architecture, Practices, and Future Outlook

This article introduces NetEase's real-time computing platform Sloth, detailing its architecture, component layers, integrated IDE, operational tooling, unified metadata management, challenges such as Kudu write amplification, and proposes a tiered real‑time data‑warehouse model with a vision for storage‑compute separation and unified batch‑stream APIs.

FlinkKafkaKudu
0 likes · 13 min read
NetEase Real-Time Computing Platform (Sloth): Architecture, Practices, and Future Outlook
Youzan Coder
Youzan Coder
Aug 26, 2020 · Mobile Development

How We Built a Real‑Time Crash Feedback Platform for Mobile Apps

This article details the design and implementation of a comprehensive crash feedback platform for mobile applications, covering the motivation behind replacing third‑party services, the system architecture using Flink, Kafka and HBase, crash interception on Android, automated grouping and assignment, version filtering, daily reporting, and future enhancements.

AndroidFlinkKafka
0 likes · 15 min read
How We Built a Real‑Time Crash Feedback Platform for Mobile Apps
Full-Stack Internet Architecture
Full-Stack Internet Architecture
Aug 26, 2020 · Backend Development

Interview Experience and Technical Q&A for a Java Backend Position at Tencent Cloud (Xi'an)

The article shares the author's recent move to Xi'an, discusses the local job market, and provides detailed interview questions and answers on Java backend topics such as Redis replication, Kafka performance, MySQL transactions, and JVM garbage collection to help job seekers prepare effectively.

Backend DevelopmentJVMJava
0 likes · 8 min read
Interview Experience and Technical Q&A for a Java Backend Position at Tencent Cloud (Xi'an)
Java Architect Essentials
Java Architect Essentials
Aug 25, 2020 · Backend Development

Understanding Kafka: Core Concepts, Architecture, and Performance Secrets

This article explains Kafka's role as a message system, details its fundamental components such as topics, partitions, producers, consumers, and replicas, describes how Zookeeper coordinates the cluster, and explores performance optimizations like sequential writes, zero‑copy, and network design.

ArchitectureBackend DevelopmentKafka
0 likes · 12 min read
Understanding Kafka: Core Concepts, Architecture, and Performance Secrets
DataFunTalk
DataFunTalk
Aug 25, 2020 · Databases

Real‑time Data Ingestion and Optimization with ClickHouse at ByteDance

This article details ByteDance's engineering practices for using ClickHouse to ingest, store, and query massive real‑time recommendation and advertising data, covering early external‑transaction mechanisms, the risks of direct INSERTs, the design and evaluation of Kafka Engine versus Flink pipelines, and a series of performance and reliability improvements implemented to support high‑frequency workloads.

ClickHouseData IngestionKafka
0 likes · 20 min read
Real‑time Data Ingestion and Optimization with ClickHouse at ByteDance
Didi Tech
Didi Tech
Aug 24, 2020 · Big Data

Evolution and Architecture of DiDi Data Channel Service

DiDi’s Data Channel Service evolved from a fragmented component system into a unified, SLA‑driven platform with a UI‑based Sync Center and Flink‑powered StreamSQL engine, dramatically improving task creation speed, resource utilization, and reliability while automating issue diagnosis for company‑wide real‑time and offline data synchronization.

ETLFlinkKafka
0 likes · 12 min read
Evolution and Architecture of DiDi Data Channel Service
Architect
Architect
Aug 21, 2020 · Backend Development

Message Queue Interview Guide: Benefits, Drawbacks, Choosing the Right MQ, and Ensuring High Availability

This article explains why and when to use message queues, outlines their advantages and disadvantages, compares popular MQ products such as Kafka, RabbitMQ, RocketMQ and ActiveMQ, and provides practical advice on high‑availability, duplicate‑consumption prevention, and idempotent design for interview preparation.

KafkaMQRabbitMQ
0 likes · 19 min read
Message Queue Interview Guide: Benefits, Drawbacks, Choosing the Right MQ, and Ensuring High Availability
Big Data Technology & Architecture
Big Data Technology & Architecture
Aug 15, 2020 · Big Data

Step-by-Step Guide to Building an ELK Stack with Kafka, Zookeeper, Logstash, and Filebeat for Log Collection

This tutorial provides a comprehensive, step-by-step procedure for setting up a log‑collection pipeline using Filebeat, Kafka, Zookeeper, Logstash, Elasticsearch, and Kibana across multiple servers, covering hardware preparation, system tuning, software installation, configuration files, and verification commands.

ELKFilebeatKafka
0 likes · 11 min read
Step-by-Step Guide to Building an ELK Stack with Kafka, Zookeeper, Logstash, and Filebeat for Log Collection
Top Architect
Top Architect
Aug 14, 2020 · Big Data

Billion‑Row MySQL to HBase Synchronization: Load Data, Kafka‑Thrift, and Flink Solutions

This article presents a comprehensive guide for transferring massive MySQL datasets to HBase, covering environment setup on Ubuntu, three synchronization methods—MySQL LOAD DATA, a Kafka‑Thrift pipeline using Maxwell, and real‑time Flink processing—along with performance comparisons and practical tips for Hadoop, HBase, Kafka, Zookeeper, Phoenix, and related tools.

DataSyncFlinkHBase
0 likes · 24 min read
Billion‑Row MySQL to HBase Synchronization: Load Data, Kafka‑Thrift, and Flink Solutions
Tencent Cloud Middleware
Tencent Cloud Middleware
Aug 12, 2020 · Big Data

How Serverless Functions Can Replace Traditional Kafka Data Pipelines for Lower Cost and Easier Scaling

This article explains how Tencent Cloud CKafka works, describes the challenges of traditional open‑source data‑flow solutions, and demonstrates a Serverless Function approach—complete with architecture diagrams and code examples—to achieve low‑cost, auto‑scaling Kafka‑to‑Elasticsearch pipelines.

CKafkaElasticsearchFunction-as-a-Service
0 likes · 12 min read
How Serverless Functions Can Replace Traditional Kafka Data Pipelines for Lower Cost and Easier Scaling
IT Architects Alliance
IT Architects Alliance
Aug 12, 2020 · Big Data

Introduction to Confluent KSQL for Real-Time Stream Processing

This article introduces Confluent KSQL, a SQL‑based real‑time stream processing engine for Kafka, covering its architecture, stream vs table concepts, query lifecycle, Docker‑based setup, DDL commands, example joins, windowed aggregations, connectors, and its advantages and limitations.

DockerKSQLKafka
0 likes · 9 min read
Introduction to Confluent KSQL for Real-Time Stream Processing
Top Architect
Top Architect
Aug 11, 2020 · Big Data

Kafka Basics and Cluster Architecture Overview

This article provides a comprehensive introduction to Kafka, covering its role as a messaging system, core concepts such as topics, partitions, producers, consumers, and messages, and then delves into the cluster architecture including replicas, consumer groups, controller coordination with Zookeeper, performance optimizations, log segmentation, and network design.

KafkaStreamingcluster architecture
0 likes · 11 min read
Kafka Basics and Cluster Architecture Overview
New Oriental Technology
New Oriental Technology
Aug 11, 2020 · Backend Development

Engineering Case Study of New Oriental Cloud Classroom Backend Architecture and Scaling During the Pandemic

The article details how New Oriental's Cloud Classroom backend, built with Java, Spring, MySQL, Redis, Kafka, Sentinel, and other modern technologies, scaled to support millions of users and a hundred‑fold surge in demand during the pandemic through architectural optimizations, distributed caching, traffic control, and rapid performance improvements.

JavaKafkaRedis
0 likes · 7 min read
Engineering Case Study of New Oriental Cloud Classroom Backend Architecture and Scaling During the Pandemic
Open Source Linux
Open Source Linux
Aug 11, 2020 · Backend Development

Build a Docker‑Based Kafka Cluster and Integrate It with Spring Boot

This guide walks you through creating a three‑node Kafka cluster with Zookeeper using Docker‑Compose, configuring the necessary YAML, launching the containers, and then integrating the cluster into a Spring Boot application by adding dependencies, setting Kafka properties, defining message, sender, and receiver classes, and testing the message flow.

DockerDocker ComposeKafka
0 likes · 6 min read
Build a Docker‑Based Kafka Cluster and Integrate It with Spring Boot
JavaEdge
JavaEdge
Aug 9, 2020 · Fundamentals

When Does Data Compression Boost System Performance? A Deep Dive into Kafka and RocketMQ

This article explains the significance of data compression, outlines when it should be applied, compares lossless algorithms, discusses segment selection, and details how Kafka and RocketMQ implement message compression to improve throughput while balancing CPU, storage, and network resources.

Algorithm SelectionKafkaPerformance Optimization
0 likes · 9 min read
When Does Data Compression Boost System Performance? A Deep Dive into Kafka and RocketMQ
Efficient Ops
Efficient Ops
Aug 4, 2020 · Operations

Mastering Filebeat: How to Collect and Ship Container Logs to Kafka

This article introduces Filebeat as a lightweight log shipper, explains its core components and processing flow, and provides step‑by‑step configuration examples for gathering container logs and forwarding them to Kafka or Elasticsearch in cloud‑native environments.

ElasticsearchFilebeatGo
0 likes · 13 min read
Mastering Filebeat: How to Collect and Ship Container Logs to Kafka
Big Data Technology & Architecture
Big Data Technology & Architecture
Aug 4, 2020 · Big Data

Manual Kafka Offset Management in Spark Streaming using createDirectStream (Java & Scala)

This article explains how to use Spark Streaming's Direct Approach with Kafka, manually manage offsets, and provides complete Java and Scala implementations—including a JavaKafkaManager class, a demo application, and a Scala KafkaManager—illustrating the creation of DirectKafkaInputDStream, offset handling, and integration with Spark.

JavaKafkaScala
0 likes · 14 min read
Manual Kafka Offset Management in Spark Streaming using createDirectStream (Java & Scala)
Efficient Ops
Efficient Ops
Aug 3, 2020 · Backend Development

Mastering Kafka Producer API: Tips, Configurations, and Common Pitfalls

This article provides a comprehensive guide to Kafka's producer API, covering core concepts, client‑side workflow, essential configurations, idempotent and transactional producers, and practical Java code examples to help developers avoid common pitfalls and optimize message publishing.

JavaKafkaProducer API
0 likes · 21 min read
Mastering Kafka Producer API: Tips, Configurations, and Common Pitfalls
JavaEdge
JavaEdge
Aug 1, 2020 · Backend Development

How to Choose the Right Message Queue: RabbitMQ vs RocketMQ vs Kafka

This guide outlines key criteria for selecting a message queue—open source, ecosystem, reliability, clustering, and performance—and compares RabbitMQ, RocketMQ, and Kafka, highlighting each system's strengths, weaknesses, and ideal use‑cases.

KafkaRabbitMQRocketMQ
0 likes · 10 min read
How to Choose the Right Message Queue: RabbitMQ vs RocketMQ vs Kafka
Full-Stack Internet Architecture
Full-Stack Internet Architecture
Jul 31, 2020 · Backend Development

Why Kafka Is Fast: Partition Parallelism, Sequential Disk Writes, Page Cache, Zero‑Copy, Batching and Compression

The article explains how Kafka achieves high throughput by using partition‑level parallelism, sequential disk writes with segment files, extensive use of the OS page cache, zero‑copy data paths, request batching and optional compression, while also discussing the underlying disk I/O principles.

BackendKafkaPartitioning
0 likes · 14 min read
Why Kafka Is Fast: Partition Parallelism, Sequential Disk Writes, Page Cache, Zero‑Copy, Batching and Compression
Tencent Cloud Developer
Tencent Cloud Developer
Jul 29, 2020 · Big Data

Case Study: Optimizing Tencent Cloud Elasticsearch for High‑Volume Game Log Analytics

To handle a gaming company's million‑QPS log stream, the team built a hot‑cold Tencent Cloud Elasticsearch cluster with ILM‑driven tiering, scaled CPU/heap, reduced shard count via shrink and replica tweaks, tuned Logstash‑Kafka pipelines, and employed COS snapshots and searchable snapshots, achieving stable performance and lower cost.

ElasticsearchILMKafka
0 likes · 29 min read
Case Study: Optimizing Tencent Cloud Elasticsearch for High‑Volume Game Log Analytics
Big Data Technology & Architecture
Big Data Technology & Architecture
Jul 25, 2020 · Big Data

Kafka Transactions, Replication Issues, HW/LEO Evolution, and Reliability Mechanisms

This article explains how Kafka implements transactions, handles under‑replicated partitions, manages high‑watermark and log‑end‑offset evolution, uses leader epochs for consistency, discusses read‑committed isolation, explains why read‑write separation is not supported, and describes delay queues, dead‑letter/retry queues, auditing, tracing, lag calculation, key metrics, and performance‑optimising design features.

DelayQueueHighWatermarkKafka
0 likes · 25 min read
Kafka Transactions, Replication Issues, HW/LEO Evolution, and Reliability Mechanisms
Big Data Technology & Architecture
Big Data Technology & Architecture
Jul 24, 2020 · Big Data

Key Concepts and Internal Mechanisms of Apache Kafka

This article provides an in‑depth overview of Apache Kafka’s internal topics, preferred replicas, partition allocation mechanisms, log directory structure, index files, offset and timestamp lookup, log retention and compaction policies, storage architecture, delayed operations, controller role, consumer rebalance process, and producer idempotence.

Consumer RebalanceIdempotenceKafka
0 likes · 18 min read
Key Concepts and Internal Mechanisms of Apache Kafka
Java Captain
Java Captain
Jul 24, 2020 · Operations

Enterprise Log Monitoring System Architecture and Implementation

To address the challenges of managing logs across hundreds of microservices in production, the article presents an enterprise log monitoring solution that centralizes collection via Filebeat, processes logs with Kafka Streams, visualizes data using Grafana and Kibana, and integrates Elastic APM for tracing and performance metrics.

ELKKafkaLog Monitoring
0 likes · 8 min read
Enterprise Log Monitoring System Architecture and Implementation
Big Data Technology & Architecture
Big Data Technology & Architecture
Jul 23, 2020 · Big Data

Comprehensive Kafka FAQ: Uses, Architecture, Offsets, and Partition Management

This article provides an extensive overview of Apache Kafka, covering its use cases, key concepts such as ISR, AR, HW, LEO, and LW, message ordering, the roles of partitioners, serializers and interceptors, producer and consumer client architecture, offset handling, multithreaded consumption, and topic partition management.

KafkaPartitioningbig data
0 likes · 16 min read
Comprehensive Kafka FAQ: Uses, Architecture, Offsets, and Partition Management
Big Data Technology & Architecture
Big Data Technology & Architecture
Jul 22, 2020 · Big Data

Kafka Architecture and Core Concepts: Producers, Brokers, and Consumers

This article explains Kafka's fundamental architecture, including the roles of producers, brokers, and consumers, key concepts such as topics, partitions, replicas, ISR, and controller, as well as detailed mechanisms of producer client structure, interceptors, serializers, partitioners, and consumer group rebalancing strategies.

Kafkabig datadistributed systems
0 likes · 22 min read
Kafka Architecture and Core Concepts: Producers, Brokers, and Consumers
Programmer DD
Programmer DD
Jul 22, 2020 · Big Data

How to Sync Billions of MySQL Records to HBase: 3 Powerful Methods Using Hadoop, Kafka, and Flink

This comprehensive guide walks you through setting up a pseudo‑distributed Hadoop environment, loading massive MySQL data with LOAD DATA, Python scripts, and multithreading, and then synchronizing the data to HBase using three approaches—Sqoop, a Kafka‑Thrift pipeline, and a real‑time Kafka‑Flink pipeline—while also comparing query performance of HBase and Phoenix.

FlinkHBaseKafka
0 likes · 28 min read
How to Sync Billions of MySQL Records to HBase: 3 Powerful Methods Using Hadoop, Kafka, and Flink
Tencent Cloud Developer
Tencent Cloud Developer
Jul 20, 2020 · Cloud Native

Tencent Eagle Eye Distributed Logging System Cloud Migration Practice

Tencent’s Eagle Eye distributed real‑time monitoring and log analysis platform was migrated to the cloud by rebuilding its LogSender and Kafka‑to‑ES components, switching to cloud CKafka and Elasticsearch, which boosted throughput fourfold, cut resource usage by about half, saved roughly 20 million RMB annually, and set the stage for further enhancements such as comprehensive monitoring and exactly‑once delivery.

Cloud-Native ArchitectureElasticsearchKafka
0 likes · 9 min read
Tencent Eagle Eye Distributed Logging System Cloud Migration Practice
Architects Research Society
Architects Research Society
Jul 19, 2020 · Backend Development

Comparing Kafka and Mosquitto for Microservice Communication

This article examines the challenges of microservice communication, explains why REST APIs are unsuitable, and compares two messaging broker solutions—Kafka and Mosquitto—highlighting their architectures, persistence, scalability, and suitability for high‑traffic, reliable event‑driven systems.

BackendKafkaMicroservices
0 likes · 6 min read
Comparing Kafka and Mosquitto for Microservice Communication
Big Data Technology & Architecture
Big Data Technology & Architecture
Jul 15, 2020 · Big Data

Root Causes and Solutions for Kafka Duplicate Consumption

This article analyzes the common causes of Kafka duplicate consumption, such as uncommitted offsets due to forced thread termination, auto‑commit settings, session timeouts, rebalancing, and slow processing, and provides practical solutions including disabling auto‑commit, adjusting consumer configurations, and using new consumer groups.

Consumer ConfigurationKafkaauto-commit
0 likes · 7 min read
Root Causes and Solutions for Kafka Duplicate Consumption
Architects Research Society
Architects Research Society
Jul 15, 2020 · Big Data

Introduction to Apache Kafka: A Distributed Streaming Platform

This article provides a comprehensive overview of Apache Kafka, explaining its distributed, fault‑tolerant architecture, horizontal scalability, disk‑based commit log, replication mechanisms, Streams API, KSQL, and why it is widely adopted as the backbone of event‑driven, high‑throughput systems.

KafkaStreamingdistributed systems
0 likes · 15 min read
Introduction to Apache Kafka: A Distributed Streaming Platform
Selected Java Interview Questions
Selected Java Interview Questions
Jul 10, 2020 · Backend Development

Message Queue Interview Questions and Technical Guide

This article provides a comprehensive overview of message queue concepts, covering usage scenarios, advantages, drawbacks, technology selection, high‑availability architectures, duplicate handling, data loss prevention, ordering guarantees, latency management, and design principles, supplemented with interview‑style questions and code examples.

ArchitectureKafkaMQ
0 likes · 20 min read
Message Queue Interview Questions and Technical Guide
Selected Java Interview Questions
Selected Java Interview Questions
Jul 8, 2020 · Backend Development

Message Queue Applications and Comparison of Common MQs (ActiveMQ, RabbitMQ, RocketMQ, Kafka)

This article explains the role of message queues in distributed systems, illustrates four typical scenarios—asynchronous processing, application decoupling, traffic shaping, and message communication—and compares the features of popular middleware such as ActiveMQ, RabbitMQ, RocketMQ, and Kafka.

KafkaRabbitMQasynchronous-processing
0 likes · 9 min read
Message Queue Applications and Comparison of Common MQs (ActiveMQ, RabbitMQ, RocketMQ, Kafka)
Big Data Technology Architecture
Big Data Technology Architecture
Jun 29, 2020 · Fundamentals

Kafka Storage Mechanism and Reliability Guarantees

This article explains Kafka's internal storage architecture—including topics, partitions, segments, .log and .index files—how data is read, and the various reliability mechanisms such as ISR/OSR, LEO/HW, producer acknowledgment levels, leader election strategies, and delivery semantics.

KafkaLeader ElectionProducer acks
0 likes · 9 min read
Kafka Storage Mechanism and Reliability Guarantees
Full-Stack Internet Architecture
Full-Stack Internet Architecture
Jun 23, 2020 · Backend Development

Common Kafka Interview Questions and Answers

This article reviews common Kafka interview questions, covering delay queues, idempotence, replica states, offsets, message ordering, and handling duplicate consumption, and includes example code for enabling idempotent producers along with explanations of time‑wheel mechanisms and practical solutions to consumer rebalance issues.

ConsumerIdempotenceKafka
0 likes · 9 min read
Common Kafka Interview Questions and Answers
Suning Technology
Suning Technology
Jun 19, 2020 · Big Data

How Suning’s Big Data Engine Powered a Record‑Breaking 618 Sale

Suning’s 618 shopping festival showcased a massive sales surge backed by its big‑data platform, which processed over 200 billion requests, handled 38.5 PB of daily data, and delivered 31.5 trillion computations, while Kafka and HBase sustained tens of millions of TPS to ensure a seamless consumer experience.

618 SaleHBaseKafka
0 likes · 5 min read
How Suning’s Big Data Engine Powered a Record‑Breaking 618 Sale
Architect
Architect
Jun 18, 2020 · Backend Development

Applying Message Queues for Decoupling in E‑commerce Architecture

The article explains why and how to use message queues to achieve low‑coupling, better performance, fault tolerance, and eventual consistency in an e‑commerce order‑processing flow, discusses common pitfalls such as message loss and duplication, and compares popular queue products like RabbitMQ, Kafka, and RocketMQ.

KafkaRabbitMQbackend architecture
0 likes · 10 min read
Applying Message Queues for Decoupling in E‑commerce Architecture
Full-Stack Internet Architecture
Full-Stack Internet Architecture
Jun 18, 2020 · Big Data

Kafka Interview Questions: High Availability, Reliability, Consistency, Performance, and Usage Rationale

This article explains common Kafka interview questions by analyzing the system's high‑availability design, reliability mechanisms, consistency model, performance tricks such as sequential writes and zero‑copy, and the reasons for using Kafka and message queues, providing both conceptual insight and practical details.

KafkaPerformanceconsistency
0 likes · 12 min read
Kafka Interview Questions: High Availability, Reliability, Consistency, Performance, and Usage Rationale
58 Tech
58 Tech
Jun 10, 2020 · Big Data

Real‑time Data Warehouse Practices at 58 Tongcheng Bao: From Spark Streaming 1.0 to Flink‑based 2.0

This article details the evolution of 58 Tongcheng Bao's real‑time data warehouse, describing the initial Spark‑Streaming architecture, its limitations, and the redesign using Flink with a layered ODS‑DWD‑DWS‑APP model, data‑quality monitoring, join techniques, and the resulting improvements in latency and accuracy.

Data QualityFlinkKafka
0 likes · 9 min read
Real‑time Data Warehouse Practices at 58 Tongcheng Bao: From Spark Streaming 1.0 to Flink‑based 2.0
Big Data Technology & Architecture
Big Data Technology & Architecture
Jun 4, 2020 · Big Data

Kafka for Data Ingestion and Event Distribution: Production‑Consumer and Publish‑Subscribe Patterns

This article explains how Kafka can be used for data ingestion and event distribution by illustrating production‑consumer and publish‑subscribe models, describing core concepts such as topics, partitions and consumer groups, and offering practical design options for handling different event scenarios.

Data IngestionEvent DistributionKafka
0 likes · 9 min read
Kafka for Data Ingestion and Event Distribution: Production‑Consumer and Publish‑Subscribe Patterns
Programmer DD
Programmer DD
Jun 4, 2020 · Backend Development

How I Traced a Sudden Data Drop After a Feature Release: 14 Debugging Steps

After a new feature caused a sharp decline in data volume, I walked through a fourteen‑step troubleshooting process—verifying the issue, inspecting code, consulting DBAs, testing locally, checking configurations, logging, packet capture, load testing, and finally identifying a Kafka partition bottleneck—to restore normal operation.

KafkaTroubleshooting
0 likes · 9 min read
How I Traced a Sudden Data Drop After a Feature Release: 14 Debugging Steps
Big Data Technology Architecture
Big Data Technology Architecture
Jun 3, 2020 · Big Data

Comprehensive Kafka Interview Questions and Answers

This article compiles essential Kafka interview topics, covering cluster sizing, partition and replica configuration, offset management, topic creation, log structure, election mechanisms, partition assignment strategies, handling data backlog, exactly‑once semantics, idempotence, transactions, and performance tuning with practical command examples.

KafkaPerformanceinterview
0 likes · 15 min read
Comprehensive Kafka Interview Questions and Answers
MaGe Linux Operations
MaGe Linux Operations
May 30, 2020 · Big Data

What Is Kafka? A Deep Dive into Distributed Streaming and Messaging

Kafka is an Apache‑hosted distributed streaming platform that provides high‑throughput, durable, publish‑subscribe messaging, originally developed by LinkedIn; this article explains its core concepts, message system classifications, architecture components, APIs, replication, consumer groups, and guarantees, comparing it with other messaging solutions.

Distributed StreamingKafkaPublish‑Subscribe
0 likes · 17 min read
What Is Kafka? A Deep Dive into Distributed Streaming and Messaging
Architects' Tech Alliance
Architects' Tech Alliance
May 27, 2020 · Fundamentals

Message Queue Overview, Models, and Comparison of ActiveMQ, RabbitMQ, RocketMQ, and Kafka

This article introduces the fundamentals of message queues, explains their characteristics, delivery models, transmission modes, and push‑pull patterns, then compares four popular implementations—ActiveMQ, RabbitMQ, RocketMQ, and Kafka—highlighting each system's strengths, weaknesses, deployment requirements, and typical use cases.

ActiveMQKafkaRabbitMQ
0 likes · 17 min read
Message Queue Overview, Models, and Comparison of ActiveMQ, RabbitMQ, RocketMQ, and Kafka
MaGe Linux Operations
MaGe Linux Operations
May 27, 2020 · Operations

Key DevOps Interview Q&A: Git, MySQL Replication, Kafka, Kubernetes

This article compiles essential DevOps interview questions covering version control differences between Git and SVN, MySQL master‑slave replication mechanics, Kafka versus traditional MQ, Kubernetes service types, pod communication, health checks, resource limits, link types, and permanent mounting techniques.

DevOpsGitKafka
0 likes · 17 min read
Key DevOps Interview Q&A: Git, MySQL Replication, Kafka, Kubernetes
Big Data Technology & Architecture
Big Data Technology & Architecture
May 24, 2020 · Big Data

Analyzing and Resolving Kafka Consumer Rebalance Errors Caused by max.poll.interval.ms

The article examines a Kafka consumer rebalance error caused by exceeding max.poll.interval.ms, explains the underlying mechanics of poll intervals, offset handling, and provides practical solutions such as adjusting max.poll.interval.ms, limiting poll records, and committing offsets per message to prevent frequent rebalances.

JavaKafkaRebalance
0 likes · 9 min read
Analyzing and Resolving Kafka Consumer Rebalance Errors Caused by max.poll.interval.ms
Laravel Tech Community
Laravel Tech Community
May 22, 2020 · Big Data

Understanding Kafka Architecture: Topics, Partitions, Consumption Model, Network and Storage

This article explains Kafka's core architecture, covering how topics and partitions are stored, the advantages of its consumption model, the internal network and threading design, and the high‑reliability distributed log storage and replication mechanisms that ensure data durability and scalability.

Consumer GroupDistributed MessagingKafka
0 likes · 11 min read
Understanding Kafka Architecture: Topics, Partitions, Consumption Model, Network and Storage
macrozheng
macrozheng
May 21, 2020 · Big Data

Mastering Kafka: Core Concepts, Architecture, and Reliability Guarantees

This comprehensive guide covers Kafka's definition, publish/subscribe model, key components, storage mechanisms, producer and consumer strategies, and reliability features such as ACK levels, ISR, and exactly‑once semantics, providing a solid foundation for real‑time big‑data processing.

KafkaStreamingbig data
0 likes · 16 min read
Mastering Kafka: Core Concepts, Architecture, and Reliability Guarantees
Big Data Technology Architecture
Big Data Technology Architecture
May 19, 2020 · Big Data

Design and Implementation of a Unified Data Lake Platform Using HBase, Kafka, and Elasticsearch

This article summarizes the design, architecture, and key modules of a company-wide data lake platform—named “Tianchi”—built on HBase, Kafka, and Elasticsearch, detailing data ingestion, strategy output, metadata management, indexing, monitoring, and offline analysis, and shares lessons learned and future plans.

ArchitectureElasticsearchHBase
0 likes · 11 min read
Design and Implementation of a Unified Data Lake Platform Using HBase, Kafka, and Elasticsearch
Selected Java Interview Questions
Selected Java Interview Questions
May 16, 2020 · Big Data

How Reddit Counts Page Views at Scale Using HyperLogLog and Kafka

The article explains Reddit's large‑scale page‑view counting system, detailing its real‑time requirements, the challenges of naive hash‑set storage, and how a hybrid approach using linear probability and HyperLogLog algorithms together with Kafka, Redis, and Cassandra achieves accurate, low‑memory, near‑real‑time analytics.

HyperLogLogKafkaReddit
0 likes · 7 min read
How Reddit Counts Page Views at Scale Using HyperLogLog and Kafka
Top Architect
Top Architect
May 14, 2020 · Big Data

Kafka Overview, Architecture, Installation, and Operational Guide

This article provides a comprehensive introduction to Kafka, covering its definition, message queue concepts, architecture components, installation steps, configuration details, startup procedures, operational commands, producer and consumer mechanisms, reliability guarantees, partition strategies, offset management, and performance optimizations.

ConsumerInstallationKafka
0 likes · 22 min read
Kafka Overview, Architecture, Installation, and Operational Guide
Architecture Digest
Architecture Digest
May 3, 2020 · Big Data

Kafka Concept Overview

This article provides a comprehensive introduction to Kafka, covering its definition, message‑queue models, architecture components, installation steps, configuration details, producer and consumer mechanisms, reliability guarantees, partition assignment strategies, offset management, and high‑performance read/write techniques.

ConsumerKafkabig data
0 likes · 20 min read
Kafka Concept Overview
Tencent Cloud Developer
Tencent Cloud Developer
Apr 28, 2020 · Big Data

Evolution of Ctrip Vacation Pricing Engine: Architecture, Challenges, and Optimizations

Ctrip’s vacation pricing engine evolved from a MySQL‑based synchronous queue to a Kafka‑driven, Spark‑parallelized architecture using HBase, dramatically cutting task generation from five hours to 1.5 hours, boosting price‑accuracy above 90 % while handling billions of calculations and external API constraints.

KafkaSparkdistributed systems
0 likes · 18 min read
Evolution of Ctrip Vacation Pricing Engine: Architecture, Challenges, and Optimizations
Tencent Cloud Developer
Tencent Cloud Developer
Apr 24, 2020 · Backend Development

Mask Reservation Mini‑Program: From Perfect Experience to Lossy Service – Architecture and Design

During the COVID‑19 pandemic, Tencent and Guangzhou built the “Suikang” mask‑reservation mini‑program in two days, handling 1.7 billion visits by shifting from real‑time inventory checks to a four‑layer “lossy” architecture—CDN caching, batch releases, Redis, Kafka queues, and asynchronous processing—to trade consistency for high availability and rapid response.

CAP theoremKafkaLossy Service
0 likes · 23 min read
Mask Reservation Mini‑Program: From Perfect Experience to Lossy Service – Architecture and Design
Big Data Technology Architecture
Big Data Technology Architecture
Apr 22, 2020 · Big Data

Key Kafka Producer Configuration Parameters and Tuning Recommendations

This article explains the most important Kafka producer configuration parameters—such as acks, max.request.size, retries, compression.type, buffer.memory, batch.size, linger.ms, request.timeout.ms, and max.in.fight.requests.per.connection—provides practical tuning advice, and presents a recommended setup to achieve high throughput without message loss.

ConfigurationKafkaMessage reliability
0 likes · 9 min read
Key Kafka Producer Configuration Parameters and Tuning Recommendations
Big Data Technology & Architecture
Big Data Technology & Architecture
Apr 19, 2020 · Big Data

Understanding the Backpressure Mechanism in Spark Streaming

This article explains Spark Streaming's backpressure mechanism, detailing how batch intervals can cause data accumulation, the role of Receivers versus DirectKafkaInputDStream, configuration to enable backpressure, and the internal workings of RateController, ReceiverRateController, ReceiverSupervisor, BlockGenerator, and rate calculations for Kafka streams.

KafkaRateControllerReceiver
0 likes · 12 min read
Understanding the Backpressure Mechanism in Spark Streaming