This article compares the main Kafka distributions—Apache Kafka, Confluent Kafka, and CDH/HDP Kafka—examining their origins, feature sets, ecosystem support, and trade‑offs to help you choose the most suitable version for your streaming workloads.
The article summarizes a 2019 Apache Flink Meetup in Shenzhen where OPPO’s big‑data platform lead explains how the company built a real‑time data warehouse using Flink SQL extensions, presents four key aspects of the evolution, application cases, and future directions.
This article describes the motivation, technology selection, implementation details, and encountered challenges of building a real‑time data warehouse using Flink, covering streaming computation, code examples, dimension‑table caching, state backend choices, and best practices for production deployment.
This article explains how to improve Spark and Spark Streaming performance by tuning serialization, broadcast variables, parallelism, batch intervals, memory usage, garbage collection, and Kafka integration, providing practical code examples and real‑world optimization results.
The article gives a thorough overview of Apache Kafka, explaining its core concepts, architecture, deployment steps, monitoring tools, and offset management, including broker and topic structures, producer/consumer APIs, replication, leader election, consumer groups, offset committing, and practical configuration and troubleshooting guidance.
The article describes how Ctrip's ticket front‑end team replaced a slow, manual log‑pulling process with a Flink‑based real‑time pipeline that streams Kafka data, indexes it in Elasticsearch, and enables second‑level log retrieval for automated scenario replay, dramatically reducing CI cycle time.
This article compares Apache Kafka and Apache Pulsar, detailing Kafka's scalability challenges, Pulsar's architectural benefits, performance gains, multi‑tenant support, security features, and provides code examples and migration guidance for large‑scale streaming applications.
With the surge in real‑time data from sensors and devices, choosing the right streaming engine is critical; this article compares Apache Spark and Apache Flink—examining their architectures, micro‑batch vs continuous processing, strengths, limitations, and use‑case suitability for Kafka‑driven pipelines.
This article explains the Google Dataflow model, covering its unified batch‑and‑stream architecture, windowing and triggering mechanisms, core primitives, time domains, and how these concepts form the foundation of modern big‑data stream processing systems.
This article presents a carefully organized catalogue of over a hundred technical posts covering Flink source‑code analysis, fundamental and advanced big‑data structures, Hadoop ecosystem components, real‑time streaming with Spark and Kafka, as well as system design guidelines and miscellaneous insights, each linked to its original publication for easy reference.
This article explains the design and implementation of Spark Structured Streaming's StateStore module, covering its distributed architecture, state sharding, versioning, batch read/write, migration, update/query APIs, maintenance compaction, and fault‑tolerance mechanisms that enable incremental continuous queries with exactly‑once guarantees.
This article explains Kafka's Exactly-Once Semantics (EOS), the role of idempotence, and how transactional support works, covering EOS semantics, producer id and sequence numbers, configuration properties, and providing Java code examples for initializing, beginning, committing, and aborting transactions.
This article explains streaming media fundamentals and compares major live‑streaming protocols—including RTMP, HTTP‑FLV, HLS, HDS, and MPEG‑DASH—detailing their architectures, workflow diagrams, MPD structure, segment handling, and practical steps for building a DASH demo with MP4Box and dash.js.
This article examines the need for real‑time computation, explains streaming versus real‑time concepts, and compares Apache Spark and Apache Flink—covering their architectures, micro‑batch and continuous processing, advantages, limitations, windowing, event‑time handling, and watermarks—to guide engine selection for Kafka‑driven workloads.
This article explores Apache Kafka’s evolution, core messaging and stream processing capabilities, typical use cases, internal storage mechanisms, API choices, and best practices for deploying Kafka on Kubernetes, providing readers with comprehensive guidance to assess suitability and implement effective Kafka solutions.
This article summarizes practical Kafka best‑practice guidelines covering hardware sizing, OS and JVM tuning, disk layout choices, replica and controller settings, broker and topic evaluation, as well as producer and consumer configuration, monitoring metrics, and strategies to prevent data loss.
This article explains Kafka message structure and offset retrieval, details Hadoop's map and reduce shuffle processes, outlines Spark's deployment modes, describes HDFS read/write mechanisms, compares reduceByKey and groupByKey performance, and discusses Spark streaming integration with Kafka and data loss prevention.
This article presents Gong Yunfei's 2019 talk from the Beijing Apache Flink x Apache Kafka conference, detailing how ByteDance leverages Kafka for its large‑scale streaming and data processing needs.
FlinkSQL's retract updates allow handling of data modifications in streaming queries by using toRetractStream, contrasting with the append-only toAppendStream mode, and this article explains the differences, when each mode applies, and provides illustrative examples and visual diagrams.
This article compares Apache Spark and Apache Flink, outlining their programming models, streaming mechanisms, state management, time semantics, and exactly‑once guarantees, and highlights the strengths and differences of each framework for batch and real‑time big‑data processing.
This article provides a comprehensive tutorial on Apache Flink's Table API, explaining its concepts, core features, and a wide range of operators such as SELECT, WHERE, GROUP BY, UNION, JOIN, and various window functions, while offering complete Scala code examples, custom sources, sinks, and an end‑to‑end job that computes page‑view counts per region using event‑time tumbling windows.
This article explains the purpose and types of SQL JOIN operators, demonstrates their syntax and semantics with examples, compares traditional database joins to Apache Flink's streaming two‑stream join implementation, and discusses optimization techniques such as state management, shuffle handling, and join reordering.
This article provides an in-depth introduction to SQL, its history and ANSI standards, then details Apache Flink's SQL capabilities—including SELECT, WHERE, GROUP BY, UNION, JOIN, window functions, and user-defined functions—accompanied by extensive code examples and a complete end‑to‑end Flink job implementation.
This article provides a step‑by‑step guide on installing Apache Kafka, creating topics, producing and consuming messages via command line, and demonstrates how to integrate Kafka with Apache Flink using the Flink‑Kafka connector, custom serialization schemas, and event‑time window processing.
This tutorial explains how to consume Kafka messages in real time using Apache Flink and persist them into a MySQL database by adding the JDBC dependency, implementing a custom RichSinkFunction, and configuring a Flink job with a Kafka source and MySQL sink.
This tutorial demonstrates how to integrate Apache Flink with Redis as a sink, showing the Maven dependency, a custom RedisMapper implementation, and a complete Flink job that reads Kafka messages, performs word count, and stores results in Redis, with plans for HBase and MySQL extensions.
This article explains the concepts of OLTP, OLAP and HTAP, describes the DRDS HTAP architecture—including its engine and storage layers, Fireworks Spark‑based engine, optimizer stages, and streaming capabilities—and demonstrates cross‑database MPP queries and streaming joins while outlining suitable use cases and limitations.
This tutorial explains how to implement a real‑time top‑N hot‑selling book ranking that outputs the most clicked books every five seconds using Apache Flink, Kafka, sliding processing‑time windows, and a custom TopN aggregation function.
This article introduces the Flink‑Kafka connector, explains essential Kafka concepts, shows how to configure checkpointing, provides Maven dependencies, and includes complete Java examples for both producing to and consuming from Kafka within a Flink streaming job.
This article explains Flink's window concepts—including time‑based, count‑based, tumbling, sliding, and session windows—provides practical Scala code examples for each type, and links to related resources on Flink basics, APIs, deployment, and advanced features.
This article introduces Apache Flink's DataSetAPI and DataStreamAPI, explains their source, transformation, and sink concepts, highlights the key differences in transformation handling, and notes the series' goal of publishing over 500 big‑data tutorials for learners from beginner to expert.
This article describes a comprehensive solution for enabling real‑time audio and video communication between existing WebRTC endpoints and WeChat Mini Programs by introducing a WebRTC Gateway and Streaming Server, detailing architecture, signaling flows, media conversion, performance optimizations, and session reliability mechanisms.
The 2018 Flink Forward China conference in Beijing showcased Apache Flink’s evolution, Alibaba’s massive contributions—including the Blink fork, real‑time BI, online learning and city‑level analytics—and highlighted how industry leaders like Alibaba, Didi and others leverage Flink for scalable, low‑latency big‑data processing across diverse use cases.
The article explains how Spark Streaming backpressure, introduced in version 1.5, automatically adjusts data ingestion rates based on processing delays, replaces manual rate limits, and details its architecture, configuration parameters, and usage for preventing data backlog and executor OOM.
By preloading the MP4 header and initial frames and routing playback through a local proxy that caches range‑requested segments in an LRU disk store, the system moves the moov box to the file start (or fetches it separately), cutting short‑video start‑up latency to roughly 800 ms and delivering near‑instant playback.
From early self‑built Storm and Spark Streaming clusters to a unified YARN‑based Spark platform and finally a low‑latency Flink system with extended CEP and StreamSQL capabilities, Didi’s real‑time computing platform evolved through three stages, delivering multi‑tenant isolation, rich SQL processing, and dramatically reduced development costs.
This talk presents Yanxuan’s real‑time data warehouse built on Flink, covering background challenges, overall architecture and implementation, data quality measures, monitoring, and practical application scenarios, while highlighting design goals of flexibility, high development efficiency, and stringent data quality requirements.
Netflix’s data‑driven strategy, which uses massive subscriber analytics to shape original content and marketing, has sparked a clash with Hollywood’s traditional, relationship‑focused approach, leading to internal power struggles, leadership changes, and a broader debate over algorithmic versus human intuition in entertainment.
This article explores Ant Financial's real-time video call platform, detailing its technical choices, system architecture, signaling reliability design, network optimization strategies, and future directions for multi‑party video conferencing and interactive live streaming.
Youku enhances online video by applying intelligent post‑production contrast mapping, device‑specific HDR tone‑mapping, high‑frame‑rate restoration through frame‑rate conversion, and ROI‑aware encoding that allocates bitrate to key visual areas, complemented by audio processing, to deliver cinema‑grade quality across diverse screens.
This article explains how Meituan tackled growing real‑time data demands by redesigning its streaming platform, adopting a layered real‑time data warehouse architecture, selecting storage and compute technologies such as Cellar, Elasticsearch, Druid and Flink, and sharing practical tips on dimension expansion, joins, and aggregation to achieve higher throughput and lower latency.
Meituan replaced its Storm‑based pipeline with a four‑layer real‑time data warehouse powered by Flink, using hybrid storage (Cellar KV, Elasticsearch, Druid, MySQL) to deliver low‑latency, high‑throughput services, dramatically simplifying SQL‑driven development, unifying metrics, cutting compute costs, and paving the way for offline‑grade accuracy and reliability.
KafkaBridge is an open‑source, multi‑language SDK built on librdkafka that offers a minimal, easy‑to‑use interface for producing and consuming messages in Apache Kafka, with optimizations for PHP‑FPM, extensive language support, and detailed performance benchmarks.
KafkaBridge is a lightweight, multi-language SDK that simplifies Kafka read/write operations, offering unified interfaces, long‑connection reuse for PHP‑FPM, and reliable message delivery, with detailed compilation steps, usage examples, and performance benchmarks across C++, Python, PHP, and Go.
Alibaba’s Ali Xiaomi prediction platform leverages AI techniques—including order and issue prediction, deep CTR models, reinforcement learning, and streaming computation—to proactively anticipate user intents, improve click‑through, resolution and satisfaction rates across multiple chatbot services, while addressing code duplication and model deployment challenges.
The article examines the evolution of big‑data processing by comparing five prevalent architectures—traditional Hadoop‑based stacks, streaming‑only designs, Kappa, Lambda, and the unified Unifield model—highlighting their strengths, weaknesses, and suitable scenarios while discussing the limitations of classic BI systems and the role of distributed storage, computation, and machine‑learning integration.
Meitu’s data platform, serving dozens of apps with 500 million monthly active users and billions of daily events, combines the Arachnia log‑collection system, Kafka ingestion, multi‑layer storage (HDFS, MongoDB, HBase, Elasticsearch), offline Hive/MapReduce processing and real‑time Storm/Flink/Naix pipelines, supported by data‑workshop tools, staged evolution for scalability, and robust security and query‑validation mechanisms.
On August 4 2018, Didi Open Source and the Open Source Database Forum hosted a salon where five industry experts presented the latest advances in open‑source databases—covering Redis multi‑data‑center deployment, MySQL InnoDB Cluster, streaming‑processing architecture, PostGIS GIS solutions, and Redis 5.0 features—followed by a Q&A, a prize draw, and a showcase of Didi’s growing open‑source portfolio.
This technical sharing session by a senior audio‑video engineer from 360 Video Cloud explains core concepts of video and audio, their encoding pipelines, media processing techniques, streaming protocols, and the challenges and key technologies behind real‑time communication (RTC).
The article reviews the ten‑year evolution of Ant Financial's message queue, detailing its core reliability, consistency, availability and performance requirements, the architectural mechanisms built to meet them, the shift to pull‑mode and API‑mode designs, and the recent integration of compute capabilities to create a smart data transmission platform.
This comprehensive guide explains Spark's ecosystem, execution principles, key features, deployment architectures, core concepts like RDD, Transformations, Actions, Jobs, Stages, Shuffle and Cache, as well as Spark Streaming mechanics and practical resource‑tuning tips for optimal big‑data processing.
This article explains how React 16 rewrote its server‑side rendering layer to support four rendering modes—including string and streaming approaches—by introducing the ReactPartialRenderer and ReactMarkupReadableStream classes, and details the process of converting virtual DOM trees into HTML strings.
This article explains the evolution of Kafka message formats from version 0.7.x through 0.8.x (including 0.9.x) to 0.10.x, detailing each field, compression handling, and the design motivations behind the changes.
This article explains the functional architecture of video players, details the multimedia engine workflow, compares web, Flash, Android, and iOS playback technologies, and addresses common challenges such as audio‑video synchronization, fast start, low latency, and buffering.
This article describes the design and implementation of a seller log system, detailing the use of Kafka for high‑throughput messaging, Storm for real‑time stream processing, Elasticsearch for hot‑data search, and HBase for cold‑data storage, along with challenges faced and optimization solutions.
iQiyi’s big playback core, created in 2013 under architect Gavin, unified fragmented players across PC, mobile and TV by evolving from a C/C++ XBMC‑based V1 to feature‑rich V3 with DRM, Dolby, hybrid P2P‑CDN, VR, multi‑instance support and major performance gains, paving the way for an intelligent next‑gen native player.
This article provides an extensive overview of Apache Spark’s ecosystem—including its data‑warehouse capabilities, ML/MLlib libraries, streaming with Spark Streaming, external frameworks, and real‑world enterprise case studies—while also noting a promotional announcement for a React Native conference.
This article introduces Apache Kafka as a fast, scalable distributed publish‑subscribe system, explains its core components, Zookeeper coordination, startup workflow, key features, and common scenarios such as log collection, activity tracking, and stream processing.
This article explains Apache Kafka’s distributed publish‑subscribe design, its core components, storage model, broker behavior, integration with ZooKeeper, performance comparisons with RabbitMQ and ActiveMQ, and provides a practical example application illustrating producer and consumer APIs.
This article details Ctrip’s redesign of its real‑time user behavior service, covering the new architecture, data flow, use of Java, Kafka, Storm, Redis, and MySQL, and how it achieves high real‑time performance, availability, scalability, and fault‑tolerance to support massive travel‑industry traffic.
This guide walks Huawei competition participants through the essential steps for building a video app, covering user‑need analysis, UI simplification, backend setup, performance considerations, screen adaptation, messaging, precise marketing, and experience measurement, while providing useful resource links.
This article explains how big data challenges traditional storage, introduces HDFS for distributed file management, describes parallel processing frameworks like MapReduce, Tez, and Spark, compares higher‑level tools such as Hive and Pig, and explores real‑time streaming and key‑value stores for low‑latency analytics.
This article outlines the three historical stages of big data technology—from early storage and batch processing, through market‑driven integration with Hive, to today’s focus on speed with Spark, Impala and streaming—while detailing the Hadoop ecosystem components such as HDFS, MapReduce, KV stores and emerging solutions like YDB.
This guide introduces St-load, a Linux‑based streaming load testing tool, and provides step‑by‑step instructions for installing, compiling, and using its RTMP, HTTP, and HLS testing features, including command‑line examples and parameter explanations for both push and pull scenarios.
This article explains the StreamSQL Application concept, its resource‑sharing and isolation mechanisms, the DDL syntax for managing Applications, and a step‑by‑step example showing how a new user can create, run, and isolate StreamJobs without affecting existing production workloads.
Tencent Analytics (TA) is a free web analytics platform that processes terabytes of daily data in real time, using a custom architecture featuring JavaScript collection, event streaming, in‑memory computation, and NoSQL storage with Redis and LevelDB, offering site owners instant insights and high availability.
This article explains the architecture, real‑time computation framework, and storage solutions of Tencent Analytics, detailing how massive TB‑level web‑traffic data are collected via JavaScript, processed in memory‑centric streaming components, and stored using Redis and LevelDB to achieve second‑level updates.
This article introduces Apache Beam, its Beam Model, and how the Beam SDK enables developers to write unified, flexible pipelines for both bounded batch jobs and unbounded streaming workloads, illustrating concepts with mobile‑gaming examples and detailed code snippets.
This article explores the evolution of live‑streaming platforms, explains common video file extensions, details compression techniques and standards such as MPEG and H.264, and highlights the bandwidth and storage challenges that drive modern video encoding decisions.
This article introduces ffmpeg as a powerful multimedia framework, explains container formats, bitrate, resolution, and frame rate concepts, outlines key live‑streaming performance metrics, and provides essential ffmpeg command‑line options and examples for streaming and transcoding.
The article explains why TCP‑based protocols cannot meet low‑latency requirements for live‑streaming conferences and introduces RTP, RTCP, jitter, round‑trip time, and three packet‑loss mitigation strategies—retransmission, forward error correction, and cross‑transport—along with a brief overview of DCCP for congestion control.
LinkedIn’s engineering team details how they have scaled Apache Kafka from billions to over a trillion daily messages, focusing on quotas, a new ZooKeeper‑free consumer, reliability enhancements, security features, monitoring frameworks, and ecosystem integrations to improve cost, availability, and performance.
LinkedIn’s engineering team details how they scaled Kafka from a few billion to over a trillion daily messages, covering quotas, a new ZooKeeper‑free consumer, reliability upgrades, security roadmaps, monitoring frameworks, failure testing, cluster balancing, and ecosystem integrations.
Apache Spark, a versatile big‑data platform supporting batch processing, SQL queries, real‑time streaming, and machine‑learning workloads, dramatically accelerates data‑intensive jobs, as demonstrated by Meituan‑Dianping, where its high‑performance engine reduces execution times and enhances scalability across diverse analytical and operational pipelines.
MySQL normally loads an entire query result into memory, which can cause out‑of‑memory errors on large tables, but by adding the -q option in the console, enabling useCursorFetch in JDBC URLs, and setting stmt.setFetchSize(Integer.MIN_VALUE), you can switch to a streaming mode that returns rows one at a time.
The article explains how Apache Kafka has become a pivotal, high‑scalable publish‑subscribe system in the big‑data ecosystem, addressing the limitations of traditional databases, enabling real‑time data integration across specialized distributed systems, and shaping future data‑governance practices.
This article explores the Bigpipe technique for accelerating first‑screen rendering by leveraging HTTP 1.1 chunked transfer, comparing implementations in PHP, Java, Node.js (including Express and Koa), and demonstrating parallel module flushing with async patterns such as callbacks, async parallel, co, and async/await.
This article provides a comprehensive technical overview of Apache Kafka, covering its architecture, core concepts, design goals, comparison with other message queues, replication, consumer groups, delivery guarantees, and performance benchmarking, making it a valuable resource for big‑data engineers.
This article compares Kafka and RabbitMQ, examining their design philosophies, throughput capabilities, consumer diversity, message ordering, and handling of individual messages, to help engineers decide which system suits high-volume or flexible-consumer scenarios and understand the trade-offs of each technology.
This article showcases seven remarkable PostgreSQL extensions—including multi‑master replication, Greenplum MPP OLAP, pg_shard/FDW sharding, PostGIS 3D GIS, GPU‑accelerated PG‑Strom, PipelineDB streaming, and the versatile FDW interface—illustrating how they enable high‑availability, massive analytics, geographic intelligence, and real‑time data processing.
This article reviews Spark's evolution from version 1.2 to 1.6, explains the DataFrame and Tungsten projects, shares Hulu’s real‑world Spark deployments, and discusses performance‑related challenges such as stack overflow, streaming receiver latency, and class‑loader deadlocks.
This article explains how to build a large‑scale, real‑time vehicle monitoring system using Apache Storm and Kafka on Alibaba Cloud, covering the challenges of big‑data ingestion, system architecture, deployment steps, performance testing, and practical lessons learned.
This article provides a comprehensive overview of Apache Spark, covering its origins, core concepts such as RDDs, transformations, actions, dependencies, execution modes, and key components like Spark SQL, Streaming, MLlib, and GraphX, while also offering practical code examples and visual illustrations.
Since 2011, LinkedIn has expanded its Kafka deployment from handling billions to over a trillion messages per day, focusing on quotas, a new ZooKeeper‑free consumer, reliability enhancements, security, monitoring frameworks, fault‑injection testing, cluster balancing, and ecosystem integrations, offering valuable lessons for large‑scale streaming systems.
Spark’s big‑data analytics ecosystem comprises core components such as the in‑memory RDD data structure, Streaming for real‑time processing, GraphX for graph analytics, MLlib for machine‑learning, Spark SQL for querying, the Tachyon file system, and SparkR, each enabling scalable, distributed computation.
This article provides a comprehensive overview of Kafka, covering its background, design goals, architecture, key terminology, message routing, consumer groups, delivery guarantees, and a comparison with other popular message queue systems such as RabbitMQ, Redis, ZeroMQ, and ActiveMQ.
