This step‑by‑step guide walks you through installing Kafka, starting a single‑node broker, producing and consuming messages, configuring a multi‑node cluster with replication, testing fault tolerance, and using Kafka Connect to import and export data between files.
This article introduces QBus, a Kafka‑derived, highly‑available message queue built for 360, detailing its origins, core features, architectural design, performance advantages over traditional queues, and Java SDK usage examples for producers and consumers.
This article explains the design and implementation of the DWS platform, detailing its log‑driven architecture with Dbus, Wormhole, and Swifts, the technical choices behind real‑time data extraction, transformation, and delivery, and real‑world use cases in finance.
The article details Umeng’s mobile big‑data platform architecture, describing its Lambda‑style hybrid design, data ingestion pipeline with dual Kafka clusters, offline and real‑time processing using Hadoop, Spark, Storm, and storage layers such as HDFS, HBase, MongoDB and Elasticsearch, while also discussing challenges in data collection, cleaning, computation, security, and value‑added services.
The paper presents a production‑grade, low‑cost mobile‑app exception monitoring platform built on Spark Streaming, Kafka, and Elasticsearch that achieves high availability through exactly‑once processing and checkpointing, minute‑level latency by decoupling raw and symbolized logs, high throughput via reservoir sampling, and dynamic scalability without code changes.
A roundup of recent tech developments covering a massive US DNS outage caused by IoT‑based DDoS attacks, Apple’s addition of ARM support to macOS Sierra, Google’s evolving 20% time policy, new multi‑data‑center features in Confluent Kafka, MariaDB’s new member, a critical OpenSSL flaw, China Mobile’s OpenStack award, and Tencent’s rapid Nexus 6P hack.
This article explains the true purpose of message queues, classifies them into broker‑based and broker‑less families, compares Kafka, RabbitMQ, and ZeroMQ in terms of performance, flexibility, and lightweight distribution, and clarifies that MQs can support both asynchronous and synchronous communication.
This article explains how a financial e‑commerce platform uses the Event Sourcing architecture pattern, an asynchronous message bus, and a Go‑based domain‑specific language to build a flexible, exactly‑once points system that decouples business rules from application code and simplifies operations.
This article presents a detailed case study of Vipshop's log platform during the 419 sales event, analyzing the 2013 architecture, bottlenecks in RabbitMQ and Storm, and the subsequent redesign using Kafka, Impala, and HBase to achieve scalable, reliable big‑data processing.
The article describes how YHD's precision recommendation team built a real‑time, highly available, traceable general recommendation platform, detailing its background, overall architecture, visual configuration and traceability subsystems, and reporting significant improvements in development speed, reuse and user satisfaction.
The article describes how Yihaodian built a low‑latency, highly available, and easily scalable streaming computation platform using Storm, Kafka, Linux containers and a custom CGroup management framework to handle massive e‑commerce traffic spikes and real‑time analytics.
This article reviews major open‑source log collection tools—Chukwa, Scribe, Flume, Logstash, Kafka, and TT—examining their architectures, strengths, and limitations to help engineers select the most suitable solution for high‑volume, low‑latency data pipelines.
This article details Zhejiang Mobile's architecture and practical experience in building a billion‑scale real‑time stream computing platform using Storm, Kafka, Flume, and Redis, covering use cases, system design, performance bottlenecks, optimization techniques, and monitoring strategies.
The article presents a detailed overview of Bilibili's GOIM live chat architecture, covering its high‑stability, high‑availability, low‑latency design, component breakdown, memory and module optimizations, network improvements, and performance testing results to achieve scalable real‑time messaging.
This article details Ctrip's journey building a unified real-time data platform—covering business motivations, architectural requirements, technology choices like Kafka and Storm, implementation of Avro schemas, monitoring, alerting, operational lessons, and future explorations such as Streaming CQL and JStorm.
The article outlines Meituan‑Dianping’s “Sensors Analytics” platform, a privately‑deployed, open‑PaaS solution that collects full‑stack user events from iOS, Android, Web and WeChat, maps IDs in near real‑time, stores detailed records in Kudu (real‑time) and Parquet (offline), and serves low‑latency queries via Impala, addressing the architectural and operational challenges of high‑throughput ingestion and data‑security requirements.
This article describes Ctrip's development of a unified real-time data platform, detailing its motivations, architectural choices such as Kafka and Storm, implementation of shared schemas, resource control, monitoring, and operational lessons, as well as experiences with Storm, JStorm, and Streaming CQL.
This article introduces a beginner-friendly architecture for real-time order analytics in a big‑data environment, detailing how Flume collects logs, Kafka buffers them, Storm processes streams, and Redis stores results, while also covering configuration, code snippets, deployment steps, and troubleshooting tips.
This article explains the internal design of Kafka's SocketServer, detailing its NIO‑based thread model with Acceptor, Processor, and Handler threads, the startup sequence, how connections are accepted and processed, and the role of RequestChannel in routing requests and responses between processors and handlers.
This article explains how Kafka's LogManager loads existing logs, manages background threads for flushing, checkpointing, cleaning, and compaction, and details the code implementations and strategies for log retention, segment cleanup, and log compression in a distributed storage environment.
The article details the design, components, and operational challenges of Youzan's unified log platform, describing its multi‑layer architecture, ingestion methods using rsyslog/logstash and Flume‑NG, Kafka‑based log center, processing pipelines with Storm/Spark, and storage in HDFS and Elasticsearch.
This article explains a practical architecture for aggregating logs from distributed Docker containers by employing Flume NG as a lightweight log collector, Kafka as a high‑throughput message bus, and custom sinks to store logs per service, module and day with low latency and minimal resource impact.
This article presents a practical architecture for centrally collecting dispersed logs from Docker‑based services in a distributed environment by leveraging Flume NG as a non‑intrusive log agent, Kafka as a high‑throughput message bus, and custom sinks to partition logs by service, module, and day.
LinkedIn’s engineering team details how they have scaled Apache Kafka from billions to over a trillion daily messages, focusing on quotas, a new ZooKeeper‑free consumer, reliability enhancements, security features, monitoring frameworks, and ecosystem integrations to improve cost, availability, and performance.
LinkedIn’s engineering team details how they scaled Kafka from a few billion to over a trillion daily messages, covering quotas, a new ZooKeeper‑free consumer, reliability upgrades, security roadmaps, monitoring frameworks, failure testing, cluster balancing, and ecosystem integrations.
This article provides a comprehensive overview of Kafka's core configuration options, detailing default values and descriptions for broker, producer, consumer, and topic‑level settings to help administrators fine‑tune performance, reliability, and resource usage.
The article describes the design, components, and implementation details of Youzan's unified log platform, covering log ingestion via rsyslog, Logstash, and Flume, centralized processing with Kafka, real‑time analysis using Storm/Spark, and storage in HDFS, Elasticsearch, and Hawk, while also discussing challenges and future improvements.
This article explains how Netflix uses the open‑source EVCache system, built on Memcached and Kafka, to provide highly reliable, low‑latency caching for its micro‑services architecture across multiple AWS regions, handling billions of objects and millions of requests per second.
This article provides a comprehensive overview of Hadoop and its surrounding ecosystem, detailing core components, storage principles, key algorithms, and a wide range of modern big‑data technologies such as Spark, Flink, Kafka, NoSQL databases, and cloud‑based processing platforms.
This article explains the JMS standard, its two messaging models (Point‑to‑Point and Publish/Subscribe), how messages are consumed synchronously or asynchronously, the core JMS programming objects, and provides an overview of common middleware such as ActiveMQ, RabbitMQ, ZeroMQ, and Kafka.
This article describes the practical architecture of Youzan's commercial e‑commerce search engine, covering data source integration, distributed real‑time indexing with Elasticsearch, Hadoop and Kafka, advanced search modules, and several performance‑tuning techniques for large‑scale deployments.
The article chronicles LinkedIn’s journey from a single‑server Leo monolith to a highly distributed, service‑oriented backend architecture, detailing the introduction of member graphs, read‑only replicas, caching layers, Kafka pipelines, Rest.li APIs, super‑blocks, and multi‑data‑center deployments to support billions of daily requests.
This article provides a comprehensive overview of Apache Flume, covering its design goals, core components, deployment architecture, configuration patterns, and step‑by‑step instructions for integrating Flume with Zookeeper and Kafka to collect and forward massive log data.
LinkedIn grew from a single monolithic Leo server handling all web requests to a complex ecosystem of over 750 independent services, employing graph databases, read replicas, caching layers, Kafka pipelines, Rest.li APIs, and multi‑data‑center deployments to support billions of daily queries.
This article presents a detailed overview of Ctrip's Hermes message queue system, covering its architectural evolution from a simple Mongo‑based design to a broker‑centric, multi‑storage solution with meta‑server coordination, and discusses practical techniques for building high‑performance, scalable messaging infrastructure.
This article presents a comprehensive Kafka benchmark using six machines to evaluate producer and consumer throughput, replication effects, message size impact, and end‑to‑end latency, providing detailed results, analysis, and reproducible test commands.
This article provides a comprehensive technical overview of Apache Kafka, covering its architecture, core concepts, design goals, comparison with other message queues, replication, consumer groups, delivery guarantees, and performance benchmarking, making it a valuable resource for big‑data engineers.
This article compares Kafka and RabbitMQ, examining their design philosophies, throughput capabilities, consumer diversity, message ordering, and handling of individual messages, to help engineers decide which system suits high-volume or flexible-consumer scenarios and understand the trade-offs of each technology.
This article introduces the fundamentals of message queues, explains common use cases such as asynchronous processing, system decoupling, traffic shaping, and log handling, and reviews popular middleware implementations including JMS, ActiveMQ, RabbitMQ, ZeroMQ, and Kafka.
Kafka persists messages on disk by organizing each topic into multiple partitions, which are further divided into segment files containing paired .index and .log files; this structure enables efficient storage, offset-based lookup, and fast retrieval of specific messages through binary search across segment indexes.
Kafka introduced a high‑availability mechanism in version 0.8 by replicating partitions across multiple brokers, designating a leader and followers, using an in‑sync replica (ISR) list to balance synchronous and asynchronous replication, and employing leader election strategies to maintain data integrity during failures.
This article explains Kafka's high‑throughput distributed messaging architecture, its typical scenarios such as log and operational data collection, the roles of producers, brokers, topics, consumers, Zookeeper, and provides a practical example of detecting abnormal user transactions.
The article presents a detailed overview of Ctrip's Hermes asynchronous messaging system, describing its architectural evolution from a simple Mongo‑based queue to a broker‑centric design with MySQL and Kafka back‑ends, and explains optimization techniques for single‑node performance, clustering, lease‑based management, and reliable delivery.
This article explains why Kafka introduced high‑availability features after version 0.8, detailing the necessity of data replication and leader election, describing Kafka’s replica distribution algorithm, replication mechanics, acknowledgment requirements, leader‑election strategies, Zookeeper structures, and the broker failover process.
Kafka, the open‑source distributed messaging system from LinkedIn, offers O(1) persistence, high throughput, partitioned topics, and flexible delivery guarantees, making it a cornerstone for modern big‑data pipelines and real‑time processing alongside Hadoop, Spark, and Storm.
This article provides a comprehensive overview of message queues, covering their core concepts, key application scenarios such as asynchronous processing, system decoupling, traffic shaping, log handling, and communication, and examines popular middleware like ActiveMQ, RabbitMQ, ZeroMQ, and Kafka, along with JMS models and programming details.
This article chronicles LinkedIn's evolution from a monolithic Leo application to a massive micro‑service ecosystem, detailing the introduction of member graphs, read‑only replicas, caching layers, Kafka pipelines, Rest.li APIs, super‑blocks, and multi‑data‑center strategies that enable handling billions of requests daily.
The article details Snapdeal's Ads platform architecture, key strategies, infrastructure, requirements, and technologies that enable a highly available, low‑latency backend capable of processing billions of daily requests with a small engineering team.
This article shares the author's practical experience building a real‑time log analysis platform at Sina, covering service scope, ELK architecture, performance optimizations, usability improvements, new features, common pitfalls, and a concise Q&A for engineers handling massive log streams.
This article explains how to build a large‑scale, real‑time vehicle monitoring system using Apache Storm and Kafka on Alibaba Cloud, covering the challenges of big‑data ingestion, system architecture, deployment steps, performance testing, and practical lessons learned.
This article explains Apache Kafka’s core concepts, high‑throughput design choices such as sequential I/O, PageCache, Sendfile, and partitioning, and provides practical performance tips and configuration recommendations for brokers, producers, and consumers in large‑scale data pipelines.
This article explains the principles of real-time computing, compares it with offline batch processing, and demonstrates a practical solution using Kafka for ingestion, Apache Storm for continuous computation, and various storage options, while also covering streaming concepts and Storm's high‑availability mechanisms.
Wacai’s architects outline six critical design decisions—including system layer separation, message passing, asynchronous processing, comprehensive data storage, robust security, and storage redundancy—that together enable a resilient, reactive financial platform capable of handling massive concurrent workloads.
The article details how LinkedIn has scaled Kafka from handling billions to trillions of messages daily, describing quota enforcement, a ZooKeeper‑free consumer, reliability enhancements, security plans, monitoring frameworks, fault‑injection testing, cluster balancing, and integration with other internal data systems.
This article explains why Kafka is essential for activity and operational data pipelines, outlines key use cases such as news feeds, relevance ranking, security, monitoring, and reporting, and details its deployment topology, design decisions, and message persistence strategies.
This article surveys the evolution of Hadoop and its ecosystem, explains core storage and processing concepts, and introduces contemporary big‑data technologies such as Spark, Flink, Kafka, Lambda architecture, NoSQL databases, and cloud‑native solutions, highlighting their roles and trade‑offs.
During a lively “Sit and Discuss” session, experts compared Spark and Hadoop, evaluated Flink against Spark, contrasted HBase with Cassandra, explained why Kafka (and sometimes Flink) is preferred for distributed messaging, and shared insights on Tachyon’s role in modern big‑data ecosystems.
Since 2011, LinkedIn has expanded its Kafka deployment from handling billions to over a trillion messages per day, focusing on quotas, a new ZooKeeper‑free consumer, reliability enhancements, security, monitoring frameworks, fault‑injection testing, cluster balancing, and ecosystem integrations, offering valuable lessons for large‑scale streaming systems.
This article outlines the design and implementation of Weidian’s end‑to‑end big data processing platform, covering dataset definition, data collection via Flume‑based DataAgent, transmission through Databus, storage options such as HDFS, Kafka and Elasticsearch, and the monitoring and resource‑integration strategies that support massive mobile commerce logs.
This article provides a concise overview of several open‑source real‑time stream processing platforms—including S4, Storm, StreamBase, HStreaming, Esper/NEsper, Kafka, Scribe, and Flume—highlighting their main features, programming languages, and project links for further reference.
This article provides an in‑depth examination of Apache Kafka’s origins, design goals, core concepts such as brokers, topics, partitions, producers and consumers, compares it with other message queues, explains its storage format, configuration options, delivery guarantees, and includes practical Java code examples for partitioning and consumption.
Kafka’s distributed log architecture stores messages in partitioned segments with indexed data files, enabling efficient sequential writes, rapid deletions, and fast offset-based lookups, as detailed through its broker, topic, partition, segment structures, file naming rules, and real‑world performance experiments.
This article explains how logs—simple, append‑only, time‑ordered records—serve as the core abstraction behind databases, distributed systems, data integration pipelines, and modern stream‑processing platforms such as Kafka and Hadoop, illustrating their design, scalability, and practical challenges.
This article explains how logs—simple, append‑only, time‑ordered records—serve as the fundamental abstraction behind databases, distributed systems, data integration pipelines, and stream‑processing platforms like Kafka and Hadoop, illustrating their role in ordering, replication, scalability, and real‑time analytics.
The article describes how a mobile analytics service processes billions of events daily using a Lambda‑style architecture that combines Kafka, Storm, Amazon EMR, and S3 to achieve scalable, fault‑tolerant batch and real‑time computation, while ensuring reliable event ingestion and graceful degradation.
This article provides a comprehensive overview of major big‑data platforms such as Hadoop, Spark, Flink, Kafka, and related ecosystem components, explaining their core concepts, storage models, processing frameworks, and architectural patterns for handling massive, distributed datasets.
This article explains how LinkedIn uses Apache Kafka as a high‑throughput, fault‑tolerant messaging backbone, detailing its architecture, message categories, layered replication, audit mechanisms, and the engineering practices that keep billions of daily messages reliable and fast.
The article explains why Kafka introduced high availability starting with version 0.8, detailing the need for data replication and leader election, describing replica distribution algorithms, replication mechanics, ISR handling, ZooKeeper structures, and the broker failover process to ensure fault‑tolerant streaming.
Pinterest built a real‑time data pipeline that streams user engagement events through Apache Kafka into Spark Streaming, enriches them with location and category information, and persists the results in MemSQL to enable fast, SQL‑based analytics for its recommendation engine.
This article provides a comprehensive overview of Apache Kafka, covering its origin, core design goals, key terminology, architectural components, message routing, consumer groups, delivery guarantees, and a detailed comparison with other popular message queue systems.
This article discusses various log collection and analysis architectures, comparing solutions such as Flume‑Kafka‑Storm pipelines, Sentry, MongoDB, ELK stack, and Hadoop, and shares practical experiences, advantages, drawbacks, and deployment tips from multiple engineers.
This article provides a comprehensive overview of Kafka, covering its background, design goals, architecture, key terminology, message routing, consumer groups, delivery guarantees, and a comparison with other popular message queue systems such as RabbitMQ, Redis, ZeroMQ, and ActiveMQ.
Kafka stores each topic as partitions that are divided into sequential segment files containing paired .log data and .index files, using global offsets and sparse memory‑mapped indexes to enable fast offset‑based lookups, efficient deletions, and minimal disk I/O in real‑world deployments.
