BestHub
Sign in
Tagged articles

Kafka

1415 articles · Page 12 of 15
Big Data Technology Architecture
Big Data Technology Architecture
Apr 15, 2020 · Big Data

Real-Time Data Warehouse Practices: Case Studies from Meituan, NetEase, Zhihu, and OPPO

This article reviews the evolution of data warehouses from traditional offline models to modern real‑time architectures, presenting detailed case studies of Meituan, NetEase, Zhihu, and OPPO, and discusses layer designs, technology choices such as Flink, Kafka, and storage options, and key lessons for building scalable real‑time warehouses.

FlinkKafkabig data
0 likes · 13 min read
Real-Time Data Warehouse Practices: Case Studies from Meituan, NetEase, Zhihu, and OPPO
Big Data Technology Architecture
Big Data Technology Architecture
Apr 13, 2020 · Backend Development

Understanding Kafka Producer: Architecture, Data Structures, Serialization, Partitioning, and Buffering

This article provides a comprehensive overview of Kafka's Producer side, covering its architecture, the ProducerRecord data structure, serialization mechanisms, partitioning logic, and the accumulator buffer, while comparing old and new Producer clients and illustrating key configurations with code examples.

AccumulatorKafkaPartitioning
0 likes · 9 min read
Understanding Kafka Producer: Architecture, Data Structures, Serialization, Partitioning, and Buffering
dbaplus Community
dbaplus Community
Apr 12, 2020 · Databases

Why and How to Migrate from MongoDB to Elasticsearch: A Practical Guide

This article explains the motivations for moving a high‑volume operation‑log system from MongoDB to Elasticsearch, outlines the existing architecture, details capacity planning, index design, and a step‑by‑step migration process using Kafka, DataX, and Spring Boot, and shares the performance gains and lessons learned.

DataXElasticsearchKafka
0 likes · 14 min read
Why and How to Migrate from MongoDB to Elasticsearch: A Practical Guide
Tencent Cloud Middleware
Tencent Cloud Middleware
Apr 9, 2020 · Operations

Scaling Kafka to Support Millions of Partitions Without Downtime

This article explains the metadata, controller, and Zookeeper challenges of supporting a million‑plus Kafka partitions and presents practical solutions such as parallel ZK fetching, metadata‑via‑topic redesign, logical cluster assembly, and physical cluster splitting to achieve large‑scale, stable Kafka deployments.

Cluster OperationsKafkaZooKeeper
0 likes · 15 min read
Scaling Kafka to Support Millions of Partitions Without Downtime
Qunar Tech Salon
Qunar Tech Salon
Apr 9, 2020 · Backend Development

RabbitMQ vs Kafka: Key Differences and How to Choose Between Them

This article compares RabbitMQ and Apache Kafka across architecture, message ordering, routing, timing, retention, fault handling, scalability, and consumer complexity, providing guidance on when to prefer each system based on functional and non‑functional requirements.

KafkaRabbitMQcomparison
0 likes · 17 min read
RabbitMQ vs Kafka: Key Differences and How to Choose Between Them
Programmer DD
Programmer DD
Mar 28, 2020 · Backend Development

Why Is Kafka So Fast? Uncover the 11 Performance Secrets

Kafka achieves its remarkable speed by combining sequential I/O, batch processing, compression, zero‑copy, careful client‑side work, and a design that avoids costly fsync and garbage collection, while maintaining durability, ordering, and at‑least‑once delivery, making it a high‑throughput, low‑latency event streaming platform.

Batch ProcessingKafkaPerformance
0 likes · 15 min read
Why Is Kafka So Fast? Uncover the 11 Performance Secrets
Programmer DD
Programmer DD
Mar 26, 2020 · Backend Development

How Zhihu Built a Scalable Long‑Connection Gateway for Real‑Time Messaging

Zhihu’s infrastructure team designed a high‑performance, scalable long‑connection gateway that decouples business logic via publish‑subscribe, leverages OpenResty, Kafka, and Redis, implements fine‑grained ACL, sliding‑window flow control, and ensures message reliability and horizontal scalability for millions of concurrent devices.

KafkaLong ConnectionMessage reliability
0 likes · 15 min read
How Zhihu Built a Scalable Long‑Connection Gateway for Real‑Time Messaging
Qunar Tech Salon
Qunar Tech Salon
Mar 19, 2020 · Big Data

Apache Kafka Overview: Architecture, Features, and Usage

This article provides a comprehensive introduction to Apache Kafka, covering its high‑throughput distributed architecture, core concepts such as topics, partitions, brokers, producers and consumers, design goals, performance characteristics, deployment steps, configuration, and example code for producers, consumers, and Spring Boot integration.

Kafkabig datadistributed systems
0 likes · 39 min read
Apache Kafka Overview: Architecture, Features, and Usage
Big Data Technology & Architecture
Big Data Technology & Architecture
Mar 13, 2020 · Backend Development

Kafka Idempotent and Transactional Messaging Overview

This article explains how Kafka implements idempotent producers and transactional messaging to achieve exactly‑once semantics, detailing the producer session identifiers, sequence numbers, broker checks, two‑phase commit workflow, consumer isolation levels, and the limitations of atomic reads.

KafkaTransactional Messagingidempotent producer
0 likes · 9 min read
Kafka Idempotent and Transactional Messaging Overview
ITFLY8 Architecture Home
ITFLY8 Architecture Home
Mar 7, 2020 · Big Data

How Kafka’s Reactor Thread Model Powers High‑Throughput Messaging

Kafka’s high‑throughput network architecture—built on NIO, a Reactor thread model, and a TCP‑based protocol—evolves from a simple synchronous processor design to a decoupled handler‑pool system, offering valuable lessons for designing scalable backend communication layers in big‑data applications.

KafkaNIOReactor Model
0 likes · 7 min read
How Kafka’s Reactor Thread Model Powers High‑Throughput Messaging
ITFLY8 Architecture Home
ITFLY8 Architecture Home
Mar 6, 2020 · Big Data

Mastering Kafka: Build Producers, Consumers, and Custom Partitioners

This extensive tutorial walks through Kafka producer and consumer fundamentals, demonstrates how to configure key‑based and custom partitioning, explains offset management, consumer group coordination, and essential configuration parameters, and includes complete Java code examples for real‑world e‑commerce scenarios.

ConsumerKafkaPartitioning
0 likes · 24 min read
Mastering Kafka: Build Producers, Consumers, and Custom Partitioners
58 Tech
58 Tech
Mar 4, 2020 · Big Data

Applying Flink State Management to Real‑Time Recommendation Scenarios

This article explains how Flink's flexible state management, including Broadcast, Keyed, and Operator states, can be used to solve real‑time recommendation challenges such as per‑minute UV, click, and exposure counting, while addressing locality mapping and data‑delay issues with Druid as the downstream store.

Broadcast StateDruidFlink
0 likes · 13 min read
Applying Flink State Management to Real‑Time Recommendation Scenarios
ITFLY8 Architecture Home
ITFLY8 Architecture Home
Mar 4, 2020 · Big Data

Understanding Kafka: Core Concepts, Architecture, and Performance Secrets

This article introduces Kafka's role as a message system, explains its fundamental components such as topics, partitions, producers, consumers, and replicas, and dives into cluster architecture, consumer groups, Zookeeper coordination, and performance optimizations like sequential writes, zero‑copy, log segmentation, and network design.

KafkaPerformancemessage queue
0 likes · 13 min read
Understanding Kafka: Core Concepts, Architecture, and Performance Secrets
dbaplus Community
dbaplus Community
Mar 3, 2020 · Big Data

How MaFengWo Scaled Kafka for Real‑Time Big Data: Lessons and Best Practices

This article details MaFengWo's practical experience with Kafka in its big‑data platform, covering three core usage scenarios, a four‑stage evolution roadmap—including version upgrades, resource isolation, security and monitoring—and future plans such as transaction‑based deduplication and consumer throttling.

KafkaResource Isolationbig data
0 likes · 17 min read
How MaFengWo Scaled Kafka for Real‑Time Big Data: Lessons and Best Practices
Mafengwo Technology
Mafengwo Technology
Feb 28, 2020 · Backend Development

How We Achieve Real‑Time MySQL‑to‑Elasticsearch Sync with Binlog and Kafka

This article explains how a large e‑commerce platform replaced a MySQL‑centric intermediate table with a binlog‑driven pipeline that streams changes through Kafka into Elasticsearch, ensuring ordered, complete, and low‑latency data synchronization while addressing schema evolution and operational monitoring.

BackendBinlogElasticsearch
0 likes · 11 min read
How We Achieve Real‑Time MySQL‑to‑Elasticsearch Sync with Binlog and Kafka
Big Data Technology Architecture
Big Data Technology Architecture
Feb 26, 2020 · Big Data

Comprehensive Guide to Kafka Architecture, Messaging Mechanisms, Replication, Controllers, and Consumer Rebalance

This article provides an in‑depth yet approachable overview of Kafka's core concepts—including its architecture, terminology, message‑sending pipeline, replication strategy, controller role, and consumer group rebalance mechanisms—helping readers quickly grasp how Kafka works as a high‑throughput distributed messaging and streaming platform.

Consumer RebalanceDistributed MessagingKafka
0 likes · 21 min read
Comprehensive Guide to Kafka Architecture, Messaging Mechanisms, Replication, Controllers, and Consumer Rebalance
Tencent Cloud Developer
Tencent Cloud Developer
Feb 18, 2020 · Backend Development

Technical Overview of Tencent Cloud CKafka for High-Scale Online Classroom Messaging

Tencent Cloud CKafka powers Tencent Classroom’s pandemic‑era online teaching by replacing a custom queue with a high‑performance, highly available, partition‑based message bus that scales to millions of real‑time interactions, offers configurable replication and tuning for reliability, and integrates with big‑data and streaming tools for analytics.

Backend DevelopmentCKafkaKafka
0 likes · 15 min read
Technical Overview of Tencent Cloud CKafka for High-Scale Online Classroom Messaging
Java Captain
Java Captain
Feb 12, 2020 · Backend Development

Comprehensive Comparison of Kafka, RabbitMQ, ZeroMQ, RocketMQ, and ActiveMQ

This article provides a detailed side‑by‑side comparison of five popular message‑queue systems—Kafka, RabbitMQ, ZeroMQ, RocketMQ and ActiveMQ—covering documentation, supported languages, protocols, storage, transactions, load balancing, clustering, management interfaces, availability, throughput, subscription models, ordering, acknowledgments, replay, retry, concurrency and more.

ActiveMQKafkaRabbitMQ
0 likes · 21 min read
Comprehensive Comparison of Kafka, RabbitMQ, ZeroMQ, RocketMQ, and ActiveMQ
Big Data Technology & Architecture
Big Data Technology & Architecture
Feb 11, 2020 · Backend Development

Message Queue Interview Guide: Benefits, Drawbacks, High Availability, Idempotency, Ordering and Design Strategies

This article provides a comprehensive interview‑oriented overview of message queues, explaining why they are used, their core advantages and disadvantages, comparing Kafka, RabbitMQ, ActiveMQ and RocketMQ, and detailing high‑availability, reliability, idempotency, ordering, backlog handling and architectural design considerations.

KafkaRabbitMQidempotency
0 likes · 33 min read
Message Queue Interview Guide: Benefits, Drawbacks, High Availability, Idempotency, Ordering and Design Strategies
Big Data Technology & Architecture
Big Data Technology & Architecture
Feb 10, 2020 · Big Data

Real‑time MySQL Binlog Capture with Canal: Principles, Architecture, Deployment and Comparison with Maxwell

This article explains how to use Alibaba's Canal to capture MySQL binlog changes in real time, covering its underlying protocol, component architecture, HA design with ZooKeeper, configuration steps, deployment examples, and a detailed comparison with alternative tools such as Maxwell and mysql_streamer.

BinlogCanalKafka
0 likes · 17 min read
Real‑time MySQL Binlog Capture with Canal: Principles, Architecture, Deployment and Comparison with Maxwell
Yanxuan Tech Team
Yanxuan Tech Team
Feb 10, 2020 · Backend Development

How Yanxuan’s Unified Message Center Scales with RocketMQ, Kafka, and K8s

This article details Yanxuan's evolution from a chaotic, multi‑queue setup to a unified, cloud‑native message center built on RocketMQ and Kafka, describing current services, scheduling mechanisms, publish‑subscribe implementation, and future plans for platformization and Kubernetes‑based resource management.

KafkaKubernetesRocketMQ
0 likes · 13 min read
How Yanxuan’s Unified Message Center Scales with RocketMQ, Kafka, and K8s
Big Data Technology Architecture
Big Data Technology Architecture
Feb 1, 2020 · Big Data

Apache Hudi 0.5.1 Release Highlights and Upgrade Guide

The Apache Hudi 0.5.1 release introduces upgraded Spark, Avro, Parquet and Kafka dependencies, new Scala support, timeline layout changes, CLI enhancements, DeltaStreamer parameter updates, Kafka offset enum revisions, key‑generator package relocation, Hive sync options, dynamic Bloom filter, bulk‑insert support, and AWS cloud storage compatibility.

Apache HudiDeltaStreamerKafka
0 likes · 6 min read
Apache Hudi 0.5.1 Release Highlights and Upgrade Guide
Architecture Digest
Architecture Digest
Jan 18, 2020 · Backend Development

Comprehensive Comparison of Kafka, RabbitMQ, ZeroMQ, RocketMQ, and ActiveMQ Across 17 Aspects

This article provides a detailed side‑by‑side comparison of five popular message‑queue systems—Kafka, RabbitMQ, ZeroMQ, RocketMQ and ActiveMQ—covering documentation, language support, protocols, storage, transactions, load balancing, clustering, management UI, availability, duplication handling, throughput, subscription models, ordering, acknowledgments, replay, retry mechanisms and concurrency.

KafkaRabbitMQmessage queue
0 likes · 25 min read
Comprehensive Comparison of Kafka, RabbitMQ, ZeroMQ, RocketMQ, and ActiveMQ Across 17 Aspects
Big Data Technology & Architecture
Big Data Technology & Architecture
Jan 16, 2020 · Big Data

Kafka Interview Guide: Core Concepts, Architecture, and Practical Tips

This article compiles essential Kafka interview material, covering its role as a message queue, usage scenarios, architectural components, storage mechanisms, consumer group rebalancing, high‑availability features, replication details, ordering guarantees, producer/consumer client design, topic management, log retention, performance optimizations, and key monitoring metrics.

KafkaReplicationbig data
0 likes · 16 min read
Kafka Interview Guide: Core Concepts, Architecture, and Practical Tips
Programmer DD
Programmer DD
Jan 13, 2020 · Backend Development

How Kafka’s Broker Controller Keeps Your Data Flowing – Inside the Replication Engine

This article dives deep into Kafka’s internal mechanics, explaining how brokers replicate data, how the controller coordinates the cluster via ZooKeeper, the roles of leader and follower replicas, ISR management, request handling, fail‑over strategies, and consumer group rebalancing, all illustrated with diagrams.

BackendBroker ControllerISR
0 likes · 36 min read
How Kafka’s Broker Controller Keeps Your Data Flowing – Inside the Replication Engine
Architect's Tech Stack
Architect's Tech Stack
Jan 12, 2020 · Backend Development

Comprehensive Guide to Spring‑Kafka Integration and Advanced Features

This article provides a systematic tutorial on using Spring‑Kafka, covering basic setup, embedded Kafka for testing, topic creation methods, message sending with KafkaTemplate, transactional messaging, request‑reply patterns, advanced @KafkaListener configurations, manual acknowledgment, listener lifecycle control, SendTo forwarding, and retry with dead‑letter queues, all illustrated with complete code examples.

@TransactionalEmbeddedKafkaKafka
0 likes · 19 min read
Comprehensive Guide to Spring‑Kafka Integration and Advanced Features
ITPUB
ITPUB
Jan 10, 2020 · Big Data

How MaFengWo Scales Kafka for Real‑Time Big Data: Lessons and Best Practices

This article details MaFengWo’s practical experience using Kafka across three core scenarios—real‑time storage, analytical data source, and business data subscription—while describing a four‑stage evolution that includes version upgrades, resource isolation, security and monitoring enhancements, and a comprehensive subscription platform, followed by future improvement plans.

Data ReplayKafkaResource Isolation
0 likes · 16 min read
How MaFengWo Scales Kafka for Real‑Time Big Data: Lessons and Best Practices
Java High-Performance Architecture
Java High-Performance Architecture
Jan 7, 2020 · Backend Development

How to Build a Scalable Reporting Service in a Microservice Architecture

To generate a user‑enriched order report in a microservice system, the article compares four approaches—direct DB access, REST data aggregation, batch pulling, and an event‑driven model—highlighting their trade‑offs in coupling, performance, scalability, and resilience, and recommends the event‑push solution.

ArchitectureData IntegrationKafka
0 likes · 5 min read
How to Build a Scalable Reporting Service in a Microservice Architecture
Big Data Technology & Architecture
Big Data Technology & Architecture
Dec 21, 2019 · Big Data

Kafka Offset Management and Replication Mechanisms Explained

This article provides a comprehensive technical overview of Kafka's offset handling, covering the request entry point, in‑memory offset sources, offset commit and fetch implementations, file storage layout, and the leader‑follower synchronization process that ensures data replication and high‑watermark updates.

High WatermarkKafkaReplication
0 likes · 16 min read
Kafka Offset Management and Replication Mechanisms Explained
Java High-Performance Architecture
Java High-Performance Architecture
Dec 17, 2019 · Backend Development

Understanding Kafka Topic Architecture: Partitions, Replication, and Failover

This article explains Kafka's topic architecture, detailing how topics are split into partitions for scalability and parallelism, the role of logs, key-based and round-robin partitioning, replication with leaders, followers, ISR, and how these mechanisms enable fault‑tolerance and high‑performance consumer failover.

BackendKafkaReplication
0 likes · 7 min read
Understanding Kafka Topic Architecture: Partitions, Replication, and Failover
Big Data Technology & Architecture
Big Data Technology & Architecture
Dec 9, 2019 · Big Data

Building a Real‑Time ETL Pipeline with Apache Flink: Kafka to HDFS with Exactly‑Once Guarantees

This article explains how to develop a real‑time ETL application using Apache Flink that reads events from Kafka, partitions them by event time into HDFS directories, and achieves exactly‑once processing through checkpointing, custom bucket assigners, and proper state backend configuration.

Apache FlinkExactly-OnceHDFS
0 likes · 11 min read
Building a Real‑Time ETL Pipeline with Apache Flink: Kafka to HDFS with Exactly‑Once Guarantees
Java High-Performance Architecture
Java High-Performance Architecture
Dec 7, 2019 · Backend Development

How Zookeeper Powers Kafka: Key Roles Explained

This article explains how Zookeeper functions as an essential part of Kafka by managing broker status, controller election, quotas, ISR tracking, node and topic registration, as well as consumer offset storage and registration, providing a comprehensive overview for interview preparation.

ConsumerKafkaZooKeeper
0 likes · 4 min read
How Zookeeper Powers Kafka: Key Roles Explained
Big Data Technology & Architecture
Big Data Technology & Architecture
Nov 24, 2019 · Big Data

Common Apache Kafka Exceptions and Their Causes

This article lists frequent Apache Kafka exceptions such as UnknownTopicOrPartitionException, LEADER_NOT_AVAILABLE, NotLeaderForPartitionException, TimeoutException, RecordTooLargeException, and others, explaining each error message, typical reasons, and practical troubleshooting steps for producers and consumers.

ConsumerKafkabig data
0 likes · 5 min read
Common Apache Kafka Exceptions and Their Causes
Architect's Tech Stack
Architect's Tech Stack
Nov 21, 2019 · Backend Development

Comprehensive Guide to Spring‑Kafka: Integration, Configuration, and Advanced Features

This article provides a thorough tutorial on using Spring‑Kafka, covering basic Maven setup, producer and consumer code, embedded Kafka for testing, programmatic topic creation, synchronous and asynchronous message sending, transaction handling, request‑reply patterns, advanced @KafkaListener options, manual acknowledgments, lifecycle control, message forwarding with @SendTo, and retry with dead‑letter queues.

@TransactionalKafkaMicroservices
0 likes · 16 min read
Comprehensive Guide to Spring‑Kafka: Integration, Configuration, and Advanced Features
G7 EasyFlow Tech Circle
G7 EasyFlow Tech Circle
Nov 21, 2019 · Big Data

How G7 Combines AI, Big Data, and IoT to Transform Logistics

This article presents a detailed overview of G7's AI‑plus‑Big‑Data‑plus‑IoT platform for logistics, describing its neutral open architecture, real‑time data pipelines using Kafka and Flink, Lambda‑style storage in HBase/Hive, and the resulting safety‑insurance and analytics capabilities.

AIFlinkIoT
0 likes · 10 min read
How G7 Combines AI, Big Data, and IoT to Transform Logistics
Java Captain
Java Captain
Nov 20, 2019 · Backend Development

Exploring Advanced Features of Spring‑Kafka: Integration, Embedded Server, Topic Management, Transactions, and Message Handling

This article provides a comprehensive guide to using Spring‑Kafka, covering simple integration, embedded Kafka for testing, creating topics programmatically, advanced KafkaTemplate usage, transactional messaging, ReplyingKafkaTemplate for request‑reply, listener configurations, manual acknowledgments, lifecycle control, message forwarding, and retry with dead‑letter queues.

Embedded KafkaJavaKafka
0 likes · 19 min read
Exploring Advanced Features of Spring‑Kafka: Integration, Embedded Server, Topic Management, Transactions, and Message Handling
Programmer DD
Programmer DD
Nov 19, 2019 · Backend Development

17‑Point Comparison: Kafka vs RabbitMQ vs ZeroMQ vs RocketMQ vs ActiveMQ

This article provides a comprehensive 17‑point comparison of five popular message queue systems—Kafka, RabbitMQ, ZeroMQ, RocketMQ, and ActiveMQ—covering documentation, supported languages, protocols, storage, transactions, load balancing, clustering, management UI, availability, message duplication, throughput, subscription models, ordering, acknowledgments, replay, retries, concurrency, and more.

ActiveMQBackendKafka
0 likes · 23 min read
17‑Point Comparison: Kafka vs RabbitMQ vs ZeroMQ vs RocketMQ vs ActiveMQ
Qunar Tech Salon
Qunar Tech Salon
Nov 18, 2019 · Databases

Data Synchronization Architecture and Refactoring for Large-Scale Travel Data at Qunar

This article describes the challenges of handling billions of travel records in Qunar's MySQL databases, compares open‑source data sync solutions like Databus and Canal, outlines the legacy system’s issues, and presents a refactored architecture that introduces Otter, ES gateway, and improved aggregation to achieve low‑latency, reliable, and scalable data synchronization.

ETLElasticsearchJava
0 likes · 19 min read
Data Synchronization Architecture and Refactoring for Large-Scale Travel Data at Qunar
Java High-Performance Architecture
Java High-Performance Architecture
Nov 12, 2019 · Backend Development

How Kafka Consumer Groups Boost Performance and Fault Tolerance

Kafka consumer groups enable multiple consumers to share partition workloads, ensuring exclusive consumption within a group, flexible consumption patterns like broadcast and unicast, and automatic fault‑tolerance through rebalancing, ultimately improving throughput, scalability, and resilience of streaming applications.

Backend DevelopmentKafkaconsumer groups
0 likes · 4 min read
How Kafka Consumer Groups Boost Performance and Fault Tolerance
Big Data Technology & Architecture
Big Data Technology & Architecture
Nov 7, 2019 · Big Data

Real‑time Dashboard with Flink: Streaming Order Data, Site Metrics, and Top‑N Merchandise Rankings

This article demonstrates how to build a one‑second‑refresh real‑time dashboard for e‑commerce order data using Apache Flink, Kafka, and Redis, covering JSON message parsing, processing‑time windows, stateful aggregation for site‑level KPIs, and efficient top‑N product ranking via Redis sorted sets.

FlinkKafkaRedis
0 likes · 11 min read
Real‑time Dashboard with Flink: Streaming Order Data, Site Metrics, and Top‑N Merchandise Rankings
DataFunTalk
DataFunTalk
Nov 7, 2019 · Big Data

Real-Time Computing Engine at Beike: Architecture, Practices, and Future Plans

This article details Beike's real‑time computing engine, covering its background, streaming platform built on Spark Streaming and Flink, data ingestion via Kafka, metadata handling, SQL‑based task development, monitoring, storage solutions, and future roadmap for resource management and AI‑enhanced monitoring.

FlinkKafkaReal-Time Computing
0 likes · 14 min read
Real-Time Computing Engine at Beike: Architecture, Practices, and Future Plans
Java Backend Technology
Java Backend Technology
Nov 6, 2019 · Backend Development

Kafka vs RabbitMQ vs ZeroMQ vs RocketMQ vs ActiveMQ: Comprehensive Feature Comparison

This article systematically compares Kafka, RabbitMQ, ZeroMQ, RocketMQ, and ActiveMQ across documentation availability, supported programming languages, protocols, storage models, transaction capabilities, load‑balancing mechanisms, clustering approaches, management interfaces, availability, duplicate handling, throughput, subscription patterns, ordering guarantees, acknowledgment strategies, replay options, retry mechanisms, and concurrency limits, helping engineers choose the right message queue for their needs.

ActiveMQKafkaMessage Queue Comparison
0 likes · 24 min read
Kafka vs RabbitMQ vs ZeroMQ vs RocketMQ vs ActiveMQ: Comprehensive Feature Comparison
Architecture Digest
Architecture Digest
Nov 5, 2019 · Big Data

Architecture Overview of Taobao, Meituan, and Didi Big Data Platforms

This article examines the big‑data architectures of three leading Chinese internet companies—Taobao, Meituan, and Didi—detailing their data sources, synchronization mechanisms, batch and streaming processing layers, and the common scheduling components that unify their Hadoop‑based ecosystems.

DidiHadoopKafka
0 likes · 7 min read
Architecture Overview of Taobao, Meituan, and Didi Big Data Platforms
Big Data Technology & Architecture
Big Data Technology & Architecture
Oct 30, 2019 · Big Data

Building a Real‑Time Data Processing Pipeline with Apache Kafka, Spark Streaming, and Cassandra

This tutorial explains how to create a highly scalable, fault‑tolerant real‑time data processing platform by configuring a Kafka topic, a Cassandra keyspace, adding Spark and connector dependencies, developing a Java‑based Spark Streaming pipeline, enabling checkpoints, and deploying the application with spark‑submit.

CassandraJavaKafka
0 likes · 8 min read
Building a Real‑Time Data Processing Pipeline with Apache Kafka, Spark Streaming, and Cassandra
DataFunTalk
DataFunTalk
Oct 25, 2019 · Big Data

Migrating Data from HBase to Kafka Using MapReduce

This article explains how to reverse the typical data flow by extracting massive Rowkeys from HBase with MapReduce, storing them on HDFS, and then using batch Get operations to retrieve the full records and write them into Kafka, while handling retries and monitoring progress.

HBaseJavaKafka
0 likes · 9 min read
Migrating Data from HBase to Kafka Using MapReduce
Architects Research Society
Architects Research Society
Oct 22, 2019 · Big Data

Continuous Delivery of Event Streaming Pipelines with Spring Cloud Data Flow

This article explains how to build, deploy, and continuously update event streaming pipelines using Spring Cloud Data Flow and Apache Kafka, covering common topologies, named destinations, parallel and partitioned streams, function composition, multiple input/output bindings, and practical shell commands for registration and management.

Continuous DeploymentEvent StreamingKafka
0 likes · 19 min read
Continuous Delivery of Event Streaming Pipelines with Spring Cloud Data Flow
dbaplus Community
dbaplus Community
Oct 20, 2019 · Big Data

Mastering Kafka: Concepts, Installation, Optimization, and Security

This comprehensive guide covers Kafka's core concepts, design principles, installation steps, configuration tweaks, performance optimizations, permission management, common operational commands, cluster scaling, log retention settings, and monitoring scripts to help you build and maintain a robust Kafka ecosystem.

ConfigurationInstallationKafka
0 likes · 20 min read
Mastering Kafka: Concepts, Installation, Optimization, and Security
JD Retail Technology
JD Retail Technology
Oct 14, 2019 · Databases

Overview of JDNoSQL Platform and Its Real-Time Advertising Use Cases

The article introduces JDNoSQL, a distributed column‑oriented key‑value store built on HDFS, outlines its core features, describes various business scenarios including real‑time ad computation, details the system architecture with Kafka and Flink, and presents table designs for ad impression and click statistics.

FlinkKafkaNoSQL
0 likes · 13 min read
Overview of JDNoSQL Platform and Its Real-Time Advertising Use Cases
UCloud Tech
UCloud Tech
Oct 11, 2019 · Big Data

Real‑Time Student Performance Analytics with Flink and Spark

This article demonstrates how to build a real‑time education analytics system by streaming answer data through Kafka into Flink or Spark, performing per‑question, per‑grade, and per‑subject aggregations, and optionally accelerating development with UFlink SQL.

Education AnalyticsFlinkKafka
0 likes · 17 min read
Real‑Time Student Performance Analytics with Flink and Spark
dbaplus Community
dbaplus Community
Oct 8, 2019 · Big Data

How to Master Large-Scale Cluster Management: 10 Real-World Troubleshooting Cases

This article shares a senior data‑platform engineer's hands‑on experience managing dozens of thousand‑node clusters, detailing nine common cluster problems and step‑by‑step solutions—including performance tuning, RPC fixes, HDFS cleanup, Hive metadata repair, Spark shuffle optimization, HBase region recovery, and Kafka bottleneck mitigation.

HBaseHadoopKafka
0 likes · 17 min read
How to Master Large-Scale Cluster Management: 10 Real-World Troubleshooting Cases
Big Data Technology & Architecture
Big Data Technology & Architecture
Sep 23, 2019 · Backend Development

Design and Evolution of Feed Stream Architecture for High‑Throughput Applications

This article analyzes the business requirements, technical challenges, and mainstream architectural solutions for large‑scale feed streams, and proposes a step‑by‑step evolution path—from a simple push model using cloud Kafka and HBase to hybrid push‑pull and recommendation‑driven designs—suitable for startups and rapidly growing platforms.

BackendHBaseKafka
0 likes · 15 min read
Design and Evolution of Feed Stream Architecture for High‑Throughput Applications
Big Data Technology & Architecture
Big Data Technology & Architecture
Sep 6, 2019 · Big Data

Big Data Development Interview Guide and Skill Tree Overview

This article provides a comprehensive interview roadmap for big data developers, outlining essential Java fundamentals, JVM internals, Linux basics, distributed theory, core frameworks such as Hadoop, Spark, Flink, Kafka, Netty, HBase, Hive, and practical algorithm topics, while also offering resume and career advice for aspiring candidates.

FlinkHadoopJava
0 likes · 15 min read
Big Data Development Interview Guide and Skill Tree Overview
DataFunTalk
DataFunTalk
Sep 5, 2019 · Big Data

Apache Beam Architecture Principles and Practical Application

This article introduces Apache Beam as a unified programming model for batch and streaming data processing, explains its architecture, core components, advantages, extensibility, and demonstrates practical usage with KafkaIO, BeamSQL, and AIoT scenarios across multiple runners.

Apache BeamData ProcessingJava
0 likes · 16 min read
Apache Beam Architecture Principles and Practical Application
dbaplus Community
dbaplus Community
Sep 4, 2019 · Operations

Running Kafka on Kubernetes: Practical Tips, Pitfalls, and Best Practices

This guide explains how to run Kafka on Kubernetes, covering runtime resource needs, storage considerations, network requirements, configuration with Pods, StatefulSets, Helm charts and Operators, performance testing, monitoring, logging, health checks, rolling updates, scaling, and backup strategies.

HelmKafkaKubernetes
0 likes · 12 min read
Running Kafka on Kubernetes: Practical Tips, Pitfalls, and Best Practices
dbaplus Community
dbaplus Community
Sep 3, 2019 · Backend Development

How We Built Real-Time MySQL-to-Elasticsearch Sync with Binlog and Kafka

To meet growing e‑commerce search demands, the team replaced a MySQL‑based intermediate table with a real‑time binlog‑driven pipeline that streams changes through Kafka into Elasticsearch, detailing design choices, ordering and completeness guarantees, custom modules, and monitoring for sub‑second sync latency.

BinlogElasticsearchKafka
0 likes · 13 min read
How We Built Real-Time MySQL-to-Elasticsearch Sync with Binlog and Kafka
HomeTech
HomeTech
Aug 15, 2019 · Big Data

Real‑Time Data Warehouse Development with Flink: Architecture, Implementation, and Lessons Learned

This article describes the motivation, technology selection, implementation details, and practical challenges of building a real‑time data warehouse using Flink, covering stream ingestion, data cleaning, dimension‑table joins, state backend choices, and operational lessons for large‑scale streaming pipelines.

FlinkKafkaReal-time Data Warehouse
0 likes · 8 min read
Real‑Time Data Warehouse Development with Flink: Architecture, Implementation, and Lessons Learned
Architecture Digest
Architecture Digest
Aug 14, 2019 · Big Data

Kafka Overview: Architecture, Storage Mechanism, Replication, and Consumer/Producer Model

Kafka is a distributed, partitioned, replicated messaging system originally developed by LinkedIn, offering high throughput, low latency, fault tolerance, and scalability; this article explains its core concepts, file storage design, partition replication, leader election, consumer groups, delivery guarantees, and operational considerations for big‑data pipelines.

KafkaReplicationbig data
0 likes · 56 min read
Kafka Overview: Architecture, Storage Mechanism, Replication, and Consumer/Producer Model
360 Quality & Efficiency
360 Quality & Efficiency
Aug 8, 2019 · Big Data

An Introduction to Kafka: Architecture, Design Principles, and Common Issues

This article introduces Kafka, covering its definition, core concepts such as topics, partitions, offsets, producers and consumers, typical use cases, underlying design principles including message‑partition allocation and retention policies, processing mechanisms, and common troubleshooting questions for real‑world deployments.

Distributed MessagingKafkabig data
0 likes · 7 min read
An Introduction to Kafka: Architecture, Design Principles, and Common Issues
Architecture Digest
Architecture Digest
Aug 8, 2019 · Big Data

Kafka Practical Guide: Concepts, Architecture, Configuration, Monitoring, and Management

This article provides a comprehensive overview of Kafka, covering its basic concepts, architecture, deployment, configuration, monitoring, producer and consumer settings, offset management, high availability, replication, leader election, and practical tips for deployment, tuning, and troubleshooting in production environments.

Kafkadistributed systemsmessage queue
0 likes · 37 min read
Kafka Practical Guide: Concepts, Architecture, Configuration, Monitoring, and Management
vivo Internet Technology
vivo Internet Technology
Aug 7, 2019 · Big Data

Understanding Apache Kafka: Concepts, Architecture, Deployment, Monitoring and Offset Management

The article gives a thorough overview of Apache Kafka, explaining its core concepts, architecture, deployment steps, monitoring tools, and offset management, including broker and topic structures, producer/consumer APIs, replication, leader election, consumer groups, offset committing, and practical configuration and troubleshooting guidance.

KafkaStreamingbig data
0 likes · 36 min read
Understanding Apache Kafka: Concepts, Architecture, Deployment, Monitoring and Offset Management
Big Data Technology Architecture
Big Data Technology Architecture
Aug 3, 2019 · Big Data

Kafka Architecture: Design Principles for High Throughput and Reliability

This article explains Kafka's design background, persistence mechanisms, disk sequential I/O optimizations, network and compression strategies, and stability features such as partitioning, replication, and ISR, illustrating how these techniques enable high‑throughput, low‑latency real‑time log processing in big‑data environments.

Kafkadisk I/Ohigh throughput
0 likes · 9 min read
Kafka Architecture: Design Principles for High Throughput and Reliability
NetEase Media Technology Team
NetEase Media Technology Team
Aug 2, 2019 · Backend Development

Delayed Message Queue Implementation: Use Cases, Comparison, and NetEase Open Course Practice

The article explains how delayed message queues replace inefficient scheduled‑task scans in distributed systems, outlines common use cases such as order timeouts and retries, compares RabbitMQ, RocketMQ, Kafka, ActiveMQ and Redis implementations, and details NetEase’s ActiveMQ‑based solution with idempotent processing and traceability.

ActiveMQKafkaRabbitMQ
0 likes · 13 min read
Delayed Message Queue Implementation: Use Cases, Comparison, and NetEase Open Course Practice