This guide demonstrates how to train a KMeans model with Transwarp Sophon and deploy it in Slipstream for real‑time streaming predictions on Kafka data, covering model export, stream creation, SQL‑based inference, and result persistence.
Overwatch is an open‑source distributed monitoring platform built by Dada‑Jingdong Home that collects, aggregates, and visualizes RPC traffic across thousands of micro‑services in real time, enabling engineers to quickly pinpoint the root cause of system failures using directed‑graph visualizations and CQRS‑based data queries.
This article provides an extensive overview of Apache Spark’s ecosystem—including its data‑warehouse capabilities, ML/MLlib libraries, streaming with Spark Streaming, external frameworks, and real‑world enterprise case studies—while also noting a promotional announcement for a React Native conference.
This article details how Vimur, a Kafka‑based real‑time data pipeline, addresses the challenges of service splitting, database sharding, data migration, and synchronization by using CDC, a unified Avro format, and a change distribution platform to support search indexing, cache refresh, and reactive architectures.
PhxQueue is a Tencent‑open‑source, Paxos‑based distributed queue that delivers at‑least‑once delivery, synchronous disk flushing, strict ordering, multi‑subscription, and high throughput, outperforming Kafka in reliability and failover scenarios while supporting massive workloads such as WeChat Pay.
This article explains Kafka’s High‑Level and Low‑Level consumer models, the semantics of Consumer Groups, the rebalance algorithm and its drawbacks, and outlines the planned redesign in Kafka 0.9.x that introduces a central Coordinator to solve herd and split‑brain issues.
KSQL is an open‑source, distributed SQL engine for Apache Kafka that enables continuous, real‑time queries on streaming data, lowering the barrier for analysts to perform stream processing, monitoring, security checks, and analytics without writing code.
PhxQueue, an open‑source, Paxos‑based distributed queue from WeChat, delivers at‑least‑once delivery, synchronous disk flushing, strict ordering, multi‑subscription, and high availability, outperforming Kafka in reliability and latency while maintaining comparable throughput, as demonstrated through detailed design, performance, and failover analyses.
Jkes is a Java‑based search framework built on Kafka and ElasticSearch that provides annotation‑driven JPA‑style mapping, REST APIs for indexing and searching, and detailed integration with Spring Boot, offering developers a complete backend solution for scalable document search.
The article presents the architecture, component design, optimizations, and reliability guarantees of Bilibili's Lancer log collection system, a Flume‑based distributed pipeline that handles both real‑time and offline data streams for billions of events daily.
This article explains why Kafka introduced High Availability in version 0.8, covering the necessity of data replication and leader election, the internal replication and ACK mechanisms, Zookeeper metadata structures, broker failover procedures, and the command‑line tools that help manage and rebalance a Kafka cluster.
This guide walks you through setting up Kafka and Zookeeper, adding Spring Integration dependencies, configuring application.yml, creating producer and consumer configurations with @Configuration and @EnableKafka, implementing a @KafkaListener, and testing the integration via a Spring MVC endpoint, while highlighting common pitfalls.
This guide walks you through installing Zookeeper and Kafka on a CentOS server, configuring essential settings, creating topics, and running producers and consumers, while highlighting common pitfalls and providing the exact commands needed for a successful deployment.
This article provides a comprehensive overview of Apache Kafka, explaining its role in real‑time data pipelines and stream processing, describing key concepts such as topics, partitions, logs, producers, consumers, replication, guarantees, and how Kafka functions as both a messaging and storage system.
This article examines the architectures of Kafka and RocketMQ, analyzes their availability and reliability mechanisms, evaluates their strengths and weaknesses, and proposes a hybrid MQ design that combines the benefits of both systems while simplifying dependencies and improving fault tolerance.
This article details Ctrip's design and implementation of a high‑throughput, low‑latency user data collection platform that leverages Java NIO, Netty, and a custom Kafka‑based messaging layer, covering architecture, encryption, compression, disaster‑recovery, performance testing, and downstream analytics products.
This article examines the architectures of Kafka and RocketMQ, analyzes their availability and reliability mechanisms, compares their strengths and weaknesses, and proposes hybrid designs and simplified MQ solutions for building robust message‑queue systems.
This article describes the design, technology selection, and performance evaluation of Ctrip's real‑time user behavior data collection platform, covering Netty‑based network handling, Kafka/Hermes messaging, encryption, compression, Avro backup, and related analytics products, with detailed feasibility analysis and benchmark results.
This article details Ctrip’s redesign of its real‑time user behavior service, covering the new architecture, data flow, use of Java, Kafka, Storm, Redis, and MySQL, and how it achieves high real‑time performance, availability, scalability, and fault‑tolerance to support massive travel‑industry traffic.
This article explains Kafka's distributed architecture, topic‑partition model, replication and ISR mechanisms, data durability settings, delivery guarantees, deduplication strategies, and presents benchmark results that illustrate how configuration choices affect throughput and latency in real‑world deployments.
This article explains Kafka's distributed architecture, its topic‑partition storage model, replication and synchronization mechanisms, reliability guarantees such as ISR and high‑watermark, and presents benchmark results that illustrate how replication factor, acks settings, and partition count affect throughput and latency.
This guide walks beginners through the entire big‑data ecosystem, covering the 4V characteristics, core open‑source frameworks, Hadoop setup, Hive and SQL on Hadoop, data ingestion and export tools, task scheduling, real‑time processing with Kafka, Storm and Spark Streaming, and an introduction to machine‑learning applications.
This article chronicles the multi‑year architectural evolution of Tongcheng Travel's push notification platform, detailing early batch‑job designs, successive redesigns using Redis, MongoDB, Kafka, Go, and .NET, and the performance, scalability, and operational improvements achieved through each major version.
This article explains Kafka’s core architecture—including brokers, topics, partitions, offsets, producer and consumer mechanics, replication, availability, consistency, persistence, performance optimizations, and Zookeeper integration—providing a comprehensive guide for building reliable distributed messaging systems.
The article describes how Ctrip rebuilt its real‑time user behavior platform using a Java‑based stack (Kafka, Storm, Redis, MySQL) to achieve millisecond‑level latency, high availability, scalable performance, and robust handling of traffic spikes, failures, and data back‑pressure.
This article describes the architecture, deployment, configuration, maintenance, and performance results of a large‑scale near real‑time logging platform built with Kafka, Flume, and Elasticsearch, highlighting practical lessons and future plans for resource‑efficient operation.
This guide walks beginners through the entire big‑data ecosystem—explaining the 4V characteristics, listing essential open‑source components, teaching Hadoop setup, Hive and SparkSQL usage, data ingestion with Sqoop, Flume and Kafka, task scheduling with Oozie, and real‑time processing with Storm and Spark Streaming.
This article explains Kafka's distributed architecture, its topic‑partition storage model, replication and ISR mechanisms, leader election, delivery guarantees, configuration for high reliability, and presents extensive benchmark results showing how replication factor, acks settings, and partition count affect throughput and latency.
This article explains Kafka's core concepts and compares producer‑consumer and publish‑subscribe models, illustrating how to apply each pattern for data ingestion and event distribution in distributed backend systems, and offers practical design alternatives when Kafka’s native capabilities fall short.
This article explains why Spark Streaming combined with Kafka can only guarantee at‑least‑once delivery, outlines the challenges of delayed and out‑of‑order events, and presents practical offline‑repair, deduplication, and output‑format techniques—including code examples—to achieve exact‑once semantics in big‑data pipelines.
The Sentinel system was built to provide real‑time, zero‑modification monitoring of airline ticket business services by consuming Tianwang logs through a Storm cluster, offering flexible rule configuration, addressing performance pitfalls, and planning future enhancements such as custom monitoring scripts and visual dashboards.
This article introduces Inceptor StreamSQL, explains its core concepts of Stream, StreamJob, and Application, and provides a step‑by‑step tutorial—from creating a Kafka source to launching a StreamJob and querying results—highlighting its ease of use and performance benefits.
This article explains various enterprise log types, recommends monitoring tools like Cacti, Zabbix, Splunk, and the ELK stack, and details architectures for handling server, application, and user‑click logs using technologies such as Logstash, Elasticsearch, Kibana, Kafka, Flume, and Spark.
This article details eBay's year‑long development of an enterprise‑grade, Kafka‑driven data transmission platform, covering its architecture, core services, monitoring and automation strategies, as well as performance tuning techniques that enable high throughput, low latency, and reliable cross‑data‑center replication.
The article describes how Qunar designed and evolved its Prism real‑time data platform—leveraging ELK, Kafka, Spark, Docker, and Mesos—to improve data collection, monitoring, and analysis, reduce deployment time, and support scalable DevOps operations across the company.
This article explores the evolution of log processing in the big‑data era, detailing how ELK Stack, Kafka, and Spark Streaming work together to provide scalable, real‑time log collection, analysis, and visualization for modern cloud‑native operations.
This article describes how Qunar’s Prism platform combines ELK, Kafka, Spark, Docker and other open‑source tools to create a real‑time data pipeline that speeds up problem localization, reduces deployment time, and improves resource utilization across development and operations teams.
TDSQL’s cloud‑based audit solution combines a three‑proxy high‑availability layer, Kafka’s O(1) persistent messaging, and a distributed audit‑server that uses consistent hashing and multi‑coroutine processing to consume data within seconds, while fault‑tolerant offsets, majority acknowledgments, and Tencent Cloud MongoDB storage ensure secure, ordered, scalable, and highly reliable audit logging.
Message queues act as a crucial middleware component in distributed systems, addressing coupling, asynchronous processing, traffic shaping, and high availability, with real-world scenarios such as asynchronous handling, decoupling, traffic throttling, logging, and communication, while reviewing popular solutions like ActiveMQ, RabbitMQ, ZeroMQ, Kafka, and JMS.
This article explains Kafka's architecture, identifies its security vulnerabilities, and presents Transwarp's Kerberos authentication and ACL-based authorization solutions, including configuration steps, code examples, and best practices for building a secure Kafka service.
Learn how to construct a simple, high‑availability real‑time stream processing platform using lightweight components such as Kafka, Zookeeper, Thrift/Avro, and optional storage like MongoDB or Elasticsearch, offering a practical alternative to heavyweight frameworks like Storm and Spark Streaming for small‑to‑medium enterprises.
This article explains Kafka’s overall cluster structure, how producers write messages to partitions, and how consumer groups read messages, covering topics, partitions, leaders, followers, offsets, and load‑balancing mechanisms in a concise technical overview.
This step‑by‑step guide walks you through installing Kafka, starting a single‑node broker, producing and consuming messages, configuring a multi‑node cluster with replication, testing fault tolerance, and using Kafka Connect to import and export data between files.
This article introduces QBus, a Kafka‑derived, highly‑available message queue built for 360, detailing its origins, core features, architectural design, performance advantages over traditional queues, and Java SDK usage examples for producers and consumers.
This article explains the design and implementation of the DWS platform, detailing its log‑driven architecture with Dbus, Wormhole, and Swifts, the technical choices behind real‑time data extraction, transformation, and delivery, and real‑world use cases in finance.
The article details Umeng’s mobile big‑data platform architecture, describing its Lambda‑style hybrid design, data ingestion pipeline with dual Kafka clusters, offline and real‑time processing using Hadoop, Spark, Storm, and storage layers such as HDFS, HBase, MongoDB and Elasticsearch, while also discussing challenges in data collection, cleaning, computation, security, and value‑added services.
The paper presents a production‑grade, low‑cost mobile‑app exception monitoring platform built on Spark Streaming, Kafka, and Elasticsearch that achieves high availability through exactly‑once processing and checkpointing, minute‑level latency by decoupling raw and symbolized logs, high throughput via reservoir sampling, and dynamic scalability without code changes.
A roundup of recent tech developments covering a massive US DNS outage caused by IoT‑based DDoS attacks, Apple’s addition of ARM support to macOS Sierra, Google’s evolving 20% time policy, new multi‑data‑center features in Confluent Kafka, MariaDB’s new member, a critical OpenSSL flaw, China Mobile’s OpenStack award, and Tencent’s rapid Nexus 6P hack.
This article explains the true purpose of message queues, classifies them into broker‑based and broker‑less families, compares Kafka, RabbitMQ, and ZeroMQ in terms of performance, flexibility, and lightweight distribution, and clarifies that MQs can support both asynchronous and synchronous communication.
This article explains how a financial e‑commerce platform uses the Event Sourcing architecture pattern, an asynchronous message bus, and a Go‑based domain‑specific language to build a flexible, exactly‑once points system that decouples business rules from application code and simplifies operations.
This article presents a detailed case study of Vipshop's log platform during the 419 sales event, analyzing the 2013 architecture, bottlenecks in RabbitMQ and Storm, and the subsequent redesign using Kafka, Impala, and HBase to achieve scalable, reliable big‑data processing.
The article describes how YHD's precision recommendation team built a real‑time, highly available, traceable general recommendation platform, detailing its background, overall architecture, visual configuration and traceability subsystems, and reporting significant improvements in development speed, reuse and user satisfaction.
The article describes how Yihaodian built a low‑latency, highly available, and easily scalable streaming computation platform using Storm, Kafka, Linux containers and a custom CGroup management framework to handle massive e‑commerce traffic spikes and real‑time analytics.
This article reviews major open‑source log collection tools—Chukwa, Scribe, Flume, Logstash, Kafka, and TT—examining their architectures, strengths, and limitations to help engineers select the most suitable solution for high‑volume, low‑latency data pipelines.
This article details Zhejiang Mobile's architecture and practical experience in building a billion‑scale real‑time stream computing platform using Storm, Kafka, Flume, and Redis, covering use cases, system design, performance bottlenecks, optimization techniques, and monitoring strategies.
The article presents a detailed overview of Bilibili's GOIM live chat architecture, covering its high‑stability, high‑availability, low‑latency design, component breakdown, memory and module optimizations, network improvements, and performance testing results to achieve scalable real‑time messaging.
This article details Ctrip's journey building a unified real-time data platform—covering business motivations, architectural requirements, technology choices like Kafka and Storm, implementation of Avro schemas, monitoring, alerting, operational lessons, and future explorations such as Streaming CQL and JStorm.
The article outlines Meituan‑Dianping’s “Sensors Analytics” platform, a privately‑deployed, open‑PaaS solution that collects full‑stack user events from iOS, Android, Web and WeChat, maps IDs in near real‑time, stores detailed records in Kudu (real‑time) and Parquet (offline), and serves low‑latency queries via Impala, addressing the architectural and operational challenges of high‑throughput ingestion and data‑security requirements.
This article describes Ctrip's development of a unified real-time data platform, detailing its motivations, architectural choices such as Kafka and Storm, implementation of shared schemas, resource control, monitoring, and operational lessons, as well as experiences with Storm, JStorm, and Streaming CQL.
This article introduces a beginner-friendly architecture for real-time order analytics in a big‑data environment, detailing how Flume collects logs, Kafka buffers them, Storm processes streams, and Redis stores results, while also covering configuration, code snippets, deployment steps, and troubleshooting tips.
This article explains the internal design of Kafka's SocketServer, detailing its NIO‑based thread model with Acceptor, Processor, and Handler threads, the startup sequence, how connections are accepted and processed, and the role of RequestChannel in routing requests and responses between processors and handlers.
This article explains how Kafka's LogManager loads existing logs, manages background threads for flushing, checkpointing, cleaning, and compaction, and details the code implementations and strategies for log retention, segment cleanup, and log compression in a distributed storage environment.
The article details the design, components, and operational challenges of Youzan's unified log platform, describing its multi‑layer architecture, ingestion methods using rsyslog/logstash and Flume‑NG, Kafka‑based log center, processing pipelines with Storm/Spark, and storage in HDFS and Elasticsearch.
This article explains a practical architecture for aggregating logs from distributed Docker containers by employing Flume NG as a lightweight log collector, Kafka as a high‑throughput message bus, and custom sinks to store logs per service, module and day with low latency and minimal resource impact.
This article presents a practical architecture for centrally collecting dispersed logs from Docker‑based services in a distributed environment by leveraging Flume NG as a non‑intrusive log agent, Kafka as a high‑throughput message bus, and custom sinks to partition logs by service, module, and day.
LinkedIn’s engineering team details how they have scaled Apache Kafka from billions to over a trillion daily messages, focusing on quotas, a new ZooKeeper‑free consumer, reliability enhancements, security features, monitoring frameworks, and ecosystem integrations to improve cost, availability, and performance.
LinkedIn’s engineering team details how they scaled Kafka from a few billion to over a trillion daily messages, covering quotas, a new ZooKeeper‑free consumer, reliability upgrades, security roadmaps, monitoring frameworks, failure testing, cluster balancing, and ecosystem integrations.
This article provides a comprehensive overview of Kafka's core configuration options, detailing default values and descriptions for broker, producer, consumer, and topic‑level settings to help administrators fine‑tune performance, reliability, and resource usage.
The article describes the design, components, and implementation details of Youzan's unified log platform, covering log ingestion via rsyslog, Logstash, and Flume, centralized processing with Kafka, real‑time analysis using Storm/Spark, and storage in HDFS, Elasticsearch, and Hawk, while also discussing challenges and future improvements.
This article explains how Netflix uses the open‑source EVCache system, built on Memcached and Kafka, to provide highly reliable, low‑latency caching for its micro‑services architecture across multiple AWS regions, handling billions of objects and millions of requests per second.
This article provides a comprehensive overview of Hadoop and its surrounding ecosystem, detailing core components, storage principles, key algorithms, and a wide range of modern big‑data technologies such as Spark, Flink, Kafka, NoSQL databases, and cloud‑based processing platforms.
This article explains the JMS standard, its two messaging models (Point‑to‑Point and Publish/Subscribe), how messages are consumed synchronously or asynchronously, the core JMS programming objects, and provides an overview of common middleware such as ActiveMQ, RabbitMQ, ZeroMQ, and Kafka.
This article describes the practical architecture of Youzan's commercial e‑commerce search engine, covering data source integration, distributed real‑time indexing with Elasticsearch, Hadoop and Kafka, advanced search modules, and several performance‑tuning techniques for large‑scale deployments.
The article chronicles LinkedIn’s journey from a single‑server Leo monolith to a highly distributed, service‑oriented backend architecture, detailing the introduction of member graphs, read‑only replicas, caching layers, Kafka pipelines, Rest.li APIs, super‑blocks, and multi‑data‑center deployments to support billions of daily requests.
This article provides a comprehensive overview of Apache Flume, covering its design goals, core components, deployment architecture, configuration patterns, and step‑by‑step instructions for integrating Flume with Zookeeper and Kafka to collect and forward massive log data.
LinkedIn grew from a single monolithic Leo server handling all web requests to a complex ecosystem of over 750 independent services, employing graph databases, read replicas, caching layers, Kafka pipelines, Rest.li APIs, and multi‑data‑center deployments to support billions of daily queries.
This article presents a detailed overview of Ctrip's Hermes message queue system, covering its architectural evolution from a simple Mongo‑based design to a broker‑centric, multi‑storage solution with meta‑server coordination, and discusses practical techniques for building high‑performance, scalable messaging infrastructure.
This article presents a comprehensive Kafka benchmark using six machines to evaluate producer and consumer throughput, replication effects, message size impact, and end‑to‑end latency, providing detailed results, analysis, and reproducible test commands.
This article provides a comprehensive technical overview of Apache Kafka, covering its architecture, core concepts, design goals, comparison with other message queues, replication, consumer groups, delivery guarantees, and performance benchmarking, making it a valuable resource for big‑data engineers.
This article compares Kafka and RabbitMQ, examining their design philosophies, throughput capabilities, consumer diversity, message ordering, and handling of individual messages, to help engineers decide which system suits high-volume or flexible-consumer scenarios and understand the trade-offs of each technology.
This article introduces the fundamentals of message queues, explains common use cases such as asynchronous processing, system decoupling, traffic shaping, and log handling, and reviews popular middleware implementations including JMS, ActiveMQ, RabbitMQ, ZeroMQ, and Kafka.
Kafka persists messages on disk by organizing each topic into multiple partitions, which are further divided into segment files containing paired .index and .log files; this structure enables efficient storage, offset-based lookup, and fast retrieval of specific messages through binary search across segment indexes.
Kafka introduced a high‑availability mechanism in version 0.8 by replicating partitions across multiple brokers, designating a leader and followers, using an in‑sync replica (ISR) list to balance synchronous and asynchronous replication, and employing leader election strategies to maintain data integrity during failures.
This article explains Kafka's high‑throughput distributed messaging architecture, its typical scenarios such as log and operational data collection, the roles of producers, brokers, topics, consumers, Zookeeper, and provides a practical example of detecting abnormal user transactions.
The article presents a detailed overview of Ctrip's Hermes asynchronous messaging system, describing its architectural evolution from a simple Mongo‑based queue to a broker‑centric design with MySQL and Kafka back‑ends, and explains optimization techniques for single‑node performance, clustering, lease‑based management, and reliable delivery.
This article explains why Kafka introduced high‑availability features after version 0.8, detailing the necessity of data replication and leader election, describing Kafka’s replica distribution algorithm, replication mechanics, acknowledgment requirements, leader‑election strategies, Zookeeper structures, and the broker failover process.
Kafka, the open‑source distributed messaging system from LinkedIn, offers O(1) persistence, high throughput, partitioned topics, and flexible delivery guarantees, making it a cornerstone for modern big‑data pipelines and real‑time processing alongside Hadoop, Spark, and Storm.
This article provides a comprehensive overview of message queues, covering their core concepts, key application scenarios such as asynchronous processing, system decoupling, traffic shaping, log handling, and communication, and examines popular middleware like ActiveMQ, RabbitMQ, ZeroMQ, and Kafka, along with JMS models and programming details.
This article chronicles LinkedIn's evolution from a monolithic Leo application to a massive micro‑service ecosystem, detailing the introduction of member graphs, read‑only replicas, caching layers, Kafka pipelines, Rest.li APIs, super‑blocks, and multi‑data‑center strategies that enable handling billions of requests daily.
The article details Snapdeal's Ads platform architecture, key strategies, infrastructure, requirements, and technologies that enable a highly available, low‑latency backend capable of processing billions of daily requests with a small engineering team.
This article shares the author's practical experience building a real‑time log analysis platform at Sina, covering service scope, ELK architecture, performance optimizations, usability improvements, new features, common pitfalls, and a concise Q&A for engineers handling massive log streams.
This article explains how to build a large‑scale, real‑time vehicle monitoring system using Apache Storm and Kafka on Alibaba Cloud, covering the challenges of big‑data ingestion, system architecture, deployment steps, performance testing, and practical lessons learned.
This article explains Apache Kafka’s core concepts, high‑throughput design choices such as sequential I/O, PageCache, Sendfile, and partitioning, and provides practical performance tips and configuration recommendations for brokers, producers, and consumers in large‑scale data pipelines.
This article explains the principles of real-time computing, compares it with offline batch processing, and demonstrates a practical solution using Kafka for ingestion, Apache Storm for continuous computation, and various storage options, while also covering streaming concepts and Storm's high‑availability mechanisms.
Wacai’s architects outline six critical design decisions—including system layer separation, message passing, asynchronous processing, comprehensive data storage, robust security, and storage redundancy—that together enable a resilient, reactive financial platform capable of handling massive concurrent workloads.
The article details how LinkedIn has scaled Kafka from handling billions to trillions of messages daily, describing quota enforcement, a ZooKeeper‑free consumer, reliability enhancements, security plans, monitoring frameworks, fault‑injection testing, cluster balancing, and integration with other internal data systems.
This article explains why Kafka is essential for activity and operational data pipelines, outlines key use cases such as news feeds, relevance ranking, security, monitoring, and reporting, and details its deployment topology, design decisions, and message persistence strategies.
This article surveys the evolution of Hadoop and its ecosystem, explains core storage and processing concepts, and introduces contemporary big‑data technologies such as Spark, Flink, Kafka, Lambda architecture, NoSQL databases, and cloud‑native solutions, highlighting their roles and trade‑offs.
During a lively “Sit and Discuss” session, experts compared Spark and Hadoop, evaluated Flink against Spark, contrasted HBase with Cassandra, explained why Kafka (and sometimes Flink) is preferred for distributed messaging, and shared insights on Tachyon’s role in modern big‑data ecosystems.
Since 2011, LinkedIn has expanded its Kafka deployment from handling billions to over a trillion messages per day, focusing on quotas, a new ZooKeeper‑free consumer, reliability enhancements, security, monitoring frameworks, fault‑injection testing, cluster balancing, and ecosystem integrations, offering valuable lessons for large‑scale streaming systems.
