BestHub
Sign in
Tagged articles

Kafka

1415 articles · Page 15 of 15
21CTO
21CTO
Sep 27, 2015 · Big Data

How Weidian Built a Scalable Big Data Platform for Mobile Commerce

This article outlines the design and implementation of Weidian’s end‑to‑end big data processing platform, covering dataset definition, data collection via Flume‑based DataAgent, transmission through Databus, storage options such as HDFS, Kafka and Elasticsearch, and the monitoring and resource‑integration strategies that support massive mobile commerce logs.

ElasticsearchFlumeHadoop
0 likes · 18 min read
How Weidian Built a Scalable Big Data Platform for Mobile Commerce
Art of Distributed System Architecture Design
Art of Distributed System Architecture Design
Sep 14, 2015 · Industry Insights

Why Kafka Dominates Distributed Messaging: Architecture, Features, and Best Practices

This article provides an in‑depth examination of Apache Kafka’s origins, design goals, core concepts such as brokers, topics, partitions, producers and consumers, compares it with other message queues, explains its storage format, configuration options, delivery guarantees, and includes practical Java code examples for partitioning and consumption.

ArchitectureJavaKafka
0 likes · 22 min read
Why Kafka Dominates Distributed Messaging: Architecture, Features, and Best Practices
21CTO
21CTO
Aug 10, 2015 · Backend Development

How Kafka’s File Storage Mechanism Achieves High Performance

Kafka’s distributed log architecture stores messages in partitioned segments with indexed data files, enabling efficient sequential writes, rapid deletions, and fast offset-based lookups, as detailed through its broker, topic, partition, segment structures, file naming rules, and real‑world performance experiments.

Kafkafile storage
0 likes · 11 min read
How Kafka’s File Storage Mechanism Achieves High Performance
Architect
Architect
Jul 6, 2015 · Big Data

Understanding Logs: The Core of Distributed Systems and Data Integration

This article explains how logs—simple, append‑only, time‑ordered records—serve as the fundamental abstraction behind databases, distributed systems, data integration pipelines, and stream‑processing platforms like Kafka and Hadoop, illustrating their role in ordering, replication, scalability, and real‑time analytics.

Data IntegrationHadoopKafka
0 likes · 48 min read
Understanding Logs: The Core of Distributed Systems and Data Integration
Art of Distributed System Architecture Design
Art of Distributed System Architecture Design
Jun 15, 2015 · Big Data

Designing a Scalable Real‑Time Mobile Analytics Platform with Kafka, Storm, and Amazon EMR

The article describes how a mobile analytics service processes billions of events daily using a Lambda‑style architecture that combines Kafka, Storm, Amazon EMR, and S3 to achieve scalable, fault‑tolerant batch and real‑time computation, while ensuring reliable event ingestion and graceful degradation.

AWSKafkaStorm
0 likes · 8 min read
Designing a Scalable Real‑Time Mobile Analytics Platform with Kafka, Storm, and Amazon EMR
MaGe Linux Operations
MaGe Linux Operations
Apr 28, 2015 · Big Data

How LinkedIn Scales Kafka to Billions of Messages Every Day

This article explains how LinkedIn uses Apache Kafka as a high‑throughput, fault‑tolerant messaging backbone, detailing its architecture, message categories, layered replication, audit mechanisms, and the engineering practices that keep billions of daily messages reliable and fast.

KafkaLinkedInbig data
0 likes · 11 min read
How LinkedIn Scales Kafka to Billions of Messages Every Day
Art of Distributed System Architecture Design
Art of Distributed System Architecture Design
Apr 28, 2015 · Big Data

Understanding Kafka High Availability: Data Replication and Leader Election

The article explains why Kafka introduced high availability starting with version 0.8, detailing the need for data replication and leader election, describing replica distribution algorithms, replication mechanics, ISR handling, ZooKeeper structures, and the broker failover process to ensure fault‑tolerant streaming.

KafkaLeader ElectionZooKeeper
0 likes · 19 min read
Understanding Kafka High Availability: Data Replication and Leader Election
Meituan Technology Team
Meituan Technology Team
Jan 14, 2015 · Big Data

Kafka File Storage Mechanism and Architecture

Kafka stores each topic as partitions that are divided into sequential segment files containing paired .log data and .index files, using global offsets and sparse memory‑mapped indexes to enable fast offset‑based lookups, efficient deletions, and minimal disk I/O in real‑world deployments.

Kafkafile storagemessage queue
0 likes · 9 min read
Kafka File Storage Mechanism and Architecture