This article explains the concepts of OLTP, OLAP and HTAP, describes the DRDS HTAP architecture—including its engine and storage layers, Fireworks Spark‑based engine, optimizer stages, and streaming capabilities—and demonstrates cross‑database MPP queries and streaming joins while outlining suitable use cases and limitations.
This tutorial explains how to implement a real‑time top‑N hot‑selling book ranking that outputs the most clicked books every five seconds using Apache Flink, Kafka, sliding processing‑time windows, and a custom TopN aggregation function.
This article introduces the Flink‑Kafka connector, explains essential Kafka concepts, shows how to configure checkpointing, provides Maven dependencies, and includes complete Java examples for both producing to and consuming from Kafka within a Flink streaming job.
This article explains Flink's window concepts—including time‑based, count‑based, tumbling, sliding, and session windows—provides practical Scala code examples for each type, and links to related resources on Flink basics, APIs, deployment, and advanced features.
This article introduces Apache Flink's DataSetAPI and DataStreamAPI, explains their source, transformation, and sink concepts, highlights the key differences in transformation handling, and notes the series' goal of publishing over 500 big‑data tutorials for learners from beginner to expert.
This article describes a comprehensive solution for enabling real‑time audio and video communication between existing WebRTC endpoints and WeChat Mini Programs by introducing a WebRTC Gateway and Streaming Server, detailing architecture, signaling flows, media conversion, performance optimizations, and session reliability mechanisms.
The 2018 Flink Forward China conference in Beijing showcased Apache Flink’s evolution, Alibaba’s massive contributions—including the Blink fork, real‑time BI, online learning and city‑level analytics—and highlighted how industry leaders like Alibaba, Didi and others leverage Flink for scalable, low‑latency big‑data processing across diverse use cases.
The article explains how Spark Streaming backpressure, introduced in version 1.5, automatically adjusts data ingestion rates based on processing delays, replaces manual rate limits, and details its architecture, configuration parameters, and usage for preventing data backlog and executor OOM.
By preloading the MP4 header and initial frames and routing playback through a local proxy that caches range‑requested segments in an LRU disk store, the system moves the moov box to the file start (or fetches it separately), cutting short‑video start‑up latency to roughly 800 ms and delivering near‑instant playback.
From early self‑built Storm and Spark Streaming clusters to a unified YARN‑based Spark platform and finally a low‑latency Flink system with extended CEP and StreamSQL capabilities, Didi’s real‑time computing platform evolved through three stages, delivering multi‑tenant isolation, rich SQL processing, and dramatically reduced development costs.
This talk presents Yanxuan’s real‑time data warehouse built on Flink, covering background challenges, overall architecture and implementation, data quality measures, monitoring, and practical application scenarios, while highlighting design goals of flexibility, high development efficiency, and stringent data quality requirements.
Netflix’s data‑driven strategy, which uses massive subscriber analytics to shape original content and marketing, has sparked a clash with Hollywood’s traditional, relationship‑focused approach, leading to internal power struggles, leadership changes, and a broader debate over algorithmic versus human intuition in entertainment.
This article explores Ant Financial's real-time video call platform, detailing its technical choices, system architecture, signaling reliability design, network optimization strategies, and future directions for multi‑party video conferencing and interactive live streaming.
Youku enhances online video by applying intelligent post‑production contrast mapping, device‑specific HDR tone‑mapping, high‑frame‑rate restoration through frame‑rate conversion, and ROI‑aware encoding that allocates bitrate to key visual areas, complemented by audio processing, to deliver cinema‑grade quality across diverse screens.
This article explains how Meituan tackled growing real‑time data demands by redesigning its streaming platform, adopting a layered real‑time data warehouse architecture, selecting storage and compute technologies such as Cellar, Elasticsearch, Druid and Flink, and sharing practical tips on dimension expansion, joins, and aggregation to achieve higher throughput and lower latency.
Meituan replaced its Storm‑based pipeline with a four‑layer real‑time data warehouse powered by Flink, using hybrid storage (Cellar KV, Elasticsearch, Druid, MySQL) to deliver low‑latency, high‑throughput services, dramatically simplifying SQL‑driven development, unifying metrics, cutting compute costs, and paving the way for offline‑grade accuracy and reliability.
KafkaBridge is an open‑source, multi‑language SDK built on librdkafka that offers a minimal, easy‑to‑use interface for producing and consuming messages in Apache Kafka, with optimizations for PHP‑FPM, extensive language support, and detailed performance benchmarks.
KafkaBridge is a lightweight, multi-language SDK that simplifies Kafka read/write operations, offering unified interfaces, long‑connection reuse for PHP‑FPM, and reliable message delivery, with detailed compilation steps, usage examples, and performance benchmarks across C++, Python, PHP, and Go.
Alibaba’s Ali Xiaomi prediction platform leverages AI techniques—including order and issue prediction, deep CTR models, reinforcement learning, and streaming computation—to proactively anticipate user intents, improve click‑through, resolution and satisfaction rates across multiple chatbot services, while addressing code duplication and model deployment challenges.
The article examines the evolution of big‑data processing by comparing five prevalent architectures—traditional Hadoop‑based stacks, streaming‑only designs, Kappa, Lambda, and the unified Unifield model—highlighting their strengths, weaknesses, and suitable scenarios while discussing the limitations of classic BI systems and the role of distributed storage, computation, and machine‑learning integration.
Meitu’s data platform, serving dozens of apps with 500 million monthly active users and billions of daily events, combines the Arachnia log‑collection system, Kafka ingestion, multi‑layer storage (HDFS, MongoDB, HBase, Elasticsearch), offline Hive/MapReduce processing and real‑time Storm/Flink/Naix pipelines, supported by data‑workshop tools, staged evolution for scalability, and robust security and query‑validation mechanisms.
On August 4 2018, Didi Open Source and the Open Source Database Forum hosted a salon where five industry experts presented the latest advances in open‑source databases—covering Redis multi‑data‑center deployment, MySQL InnoDB Cluster, streaming‑processing architecture, PostGIS GIS solutions, and Redis 5.0 features—followed by a Q&A, a prize draw, and a showcase of Didi’s growing open‑source portfolio.
This technical sharing session by a senior audio‑video engineer from 360 Video Cloud explains core concepts of video and audio, their encoding pipelines, media processing techniques, streaming protocols, and the challenges and key technologies behind real‑time communication (RTC).
The article reviews the ten‑year evolution of Ant Financial's message queue, detailing its core reliability, consistency, availability and performance requirements, the architectural mechanisms built to meet them, the shift to pull‑mode and API‑mode designs, and the recent integration of compute capabilities to create a smart data transmission platform.
This comprehensive guide explains Spark's ecosystem, execution principles, key features, deployment architectures, core concepts like RDD, Transformations, Actions, Jobs, Stages, Shuffle and Cache, as well as Spark Streaming mechanics and practical resource‑tuning tips for optimal big‑data processing.
This article explains how React 16 rewrote its server‑side rendering layer to support four rendering modes—including string and streaming approaches—by introducing the ReactPartialRenderer and ReactMarkupReadableStream classes, and details the process of converting virtual DOM trees into HTML strings.
This article explains the evolution of Kafka message formats from version 0.7.x through 0.8.x (including 0.9.x) to 0.10.x, detailing each field, compression handling, and the design motivations behind the changes.
This article explains the functional architecture of video players, details the multimedia engine workflow, compares web, Flash, Android, and iOS playback technologies, and addresses common challenges such as audio‑video synchronization, fast start, low latency, and buffering.
This article describes the design and implementation of a seller log system, detailing the use of Kafka for high‑throughput messaging, Storm for real‑time stream processing, Elasticsearch for hot‑data search, and HBase for cold‑data storage, along with challenges faced and optimization solutions.
iQiyi’s big playback core, created in 2013 under architect Gavin, unified fragmented players across PC, mobile and TV by evolving from a C/C++ XBMC‑based V1 to feature‑rich V3 with DRM, Dolby, hybrid P2P‑CDN, VR, multi‑instance support and major performance gains, paving the way for an intelligent next‑gen native player.
This article provides an extensive overview of Apache Spark’s ecosystem—including its data‑warehouse capabilities, ML/MLlib libraries, streaming with Spark Streaming, external frameworks, and real‑world enterprise case studies—while also noting a promotional announcement for a React Native conference.
This article introduces Apache Kafka as a fast, scalable distributed publish‑subscribe system, explains its core components, Zookeeper coordination, startup workflow, key features, and common scenarios such as log collection, activity tracking, and stream processing.
This article explains Apache Kafka’s distributed publish‑subscribe design, its core components, storage model, broker behavior, integration with ZooKeeper, performance comparisons with RabbitMQ and ActiveMQ, and provides a practical example application illustrating producer and consumer APIs.
This article details Ctrip’s redesign of its real‑time user behavior service, covering the new architecture, data flow, use of Java, Kafka, Storm, Redis, and MySQL, and how it achieves high real‑time performance, availability, scalability, and fault‑tolerance to support massive travel‑industry traffic.
This guide walks Huawei competition participants through the essential steps for building a video app, covering user‑need analysis, UI simplification, backend setup, performance considerations, screen adaptation, messaging, precise marketing, and experience measurement, while providing useful resource links.
This article explains how big data challenges traditional storage, introduces HDFS for distributed file management, describes parallel processing frameworks like MapReduce, Tez, and Spark, compares higher‑level tools such as Hive and Pig, and explores real‑time streaming and key‑value stores for low‑latency analytics.
This article outlines the three historical stages of big data technology—from early storage and batch processing, through market‑driven integration with Hive, to today’s focus on speed with Spark, Impala and streaming—while detailing the Hadoop ecosystem components such as HDFS, MapReduce, KV stores and emerging solutions like YDB.
This guide introduces St-load, a Linux‑based streaming load testing tool, and provides step‑by‑step instructions for installing, compiling, and using its RTMP, HTTP, and HLS testing features, including command‑line examples and parameter explanations for both push and pull scenarios.
This article explains the StreamSQL Application concept, its resource‑sharing and isolation mechanisms, the DDL syntax for managing Applications, and a step‑by‑step example showing how a new user can create, run, and isolate StreamJobs without affecting existing production workloads.
Tencent Analytics (TA) is a free web analytics platform that processes terabytes of daily data in real time, using a custom architecture featuring JavaScript collection, event streaming, in‑memory computation, and NoSQL storage with Redis and LevelDB, offering site owners instant insights and high availability.
This article explains the architecture, real‑time computation framework, and storage solutions of Tencent Analytics, detailing how massive TB‑level web‑traffic data are collected via JavaScript, processed in memory‑centric streaming components, and stored using Redis and LevelDB to achieve second‑level updates.
This article introduces Apache Beam, its Beam Model, and how the Beam SDK enables developers to write unified, flexible pipelines for both bounded batch jobs and unbounded streaming workloads, illustrating concepts with mobile‑gaming examples and detailed code snippets.
This article explores the evolution of live‑streaming platforms, explains common video file extensions, details compression techniques and standards such as MPEG and H.264, and highlights the bandwidth and storage challenges that drive modern video encoding decisions.
This article introduces ffmpeg as a powerful multimedia framework, explains container formats, bitrate, resolution, and frame rate concepts, outlines key live‑streaming performance metrics, and provides essential ffmpeg command‑line options and examples for streaming and transcoding.
The article explains why TCP‑based protocols cannot meet low‑latency requirements for live‑streaming conferences and introduces RTP, RTCP, jitter, round‑trip time, and three packet‑loss mitigation strategies—retransmission, forward error correction, and cross‑transport—along with a brief overview of DCCP for congestion control.
LinkedIn’s engineering team details how they have scaled Apache Kafka from billions to over a trillion daily messages, focusing on quotas, a new ZooKeeper‑free consumer, reliability enhancements, security features, monitoring frameworks, and ecosystem integrations to improve cost, availability, and performance.
LinkedIn’s engineering team details how they scaled Kafka from a few billion to over a trillion daily messages, covering quotas, a new ZooKeeper‑free consumer, reliability upgrades, security roadmaps, monitoring frameworks, failure testing, cluster balancing, and ecosystem integrations.
Apache Spark, a versatile big‑data platform supporting batch processing, SQL queries, real‑time streaming, and machine‑learning workloads, dramatically accelerates data‑intensive jobs, as demonstrated by Meituan‑Dianping, where its high‑performance engine reduces execution times and enhances scalability across diverse analytical and operational pipelines.
MySQL normally loads an entire query result into memory, which can cause out‑of‑memory errors on large tables, but by adding the -q option in the console, enabling useCursorFetch in JDBC URLs, and setting stmt.setFetchSize(Integer.MIN_VALUE), you can switch to a streaming mode that returns rows one at a time.
The article explains how Apache Kafka has become a pivotal, high‑scalable publish‑subscribe system in the big‑data ecosystem, addressing the limitations of traditional databases, enabling real‑time data integration across specialized distributed systems, and shaping future data‑governance practices.
This article explores the Bigpipe technique for accelerating first‑screen rendering by leveraging HTTP 1.1 chunked transfer, comparing implementations in PHP, Java, Node.js (including Express and Koa), and demonstrating parallel module flushing with async patterns such as callbacks, async parallel, co, and async/await.
This article provides a comprehensive technical overview of Apache Kafka, covering its architecture, core concepts, design goals, comparison with other message queues, replication, consumer groups, delivery guarantees, and performance benchmarking, making it a valuable resource for big‑data engineers.
This article compares Kafka and RabbitMQ, examining their design philosophies, throughput capabilities, consumer diversity, message ordering, and handling of individual messages, to help engineers decide which system suits high-volume or flexible-consumer scenarios and understand the trade-offs of each technology.
This article showcases seven remarkable PostgreSQL extensions—including multi‑master replication, Greenplum MPP OLAP, pg_shard/FDW sharding, PostGIS 3D GIS, GPU‑accelerated PG‑Strom, PipelineDB streaming, and the versatile FDW interface—illustrating how they enable high‑availability, massive analytics, geographic intelligence, and real‑time data processing.
This article reviews Spark's evolution from version 1.2 to 1.6, explains the DataFrame and Tungsten projects, shares Hulu’s real‑world Spark deployments, and discusses performance‑related challenges such as stack overflow, streaming receiver latency, and class‑loader deadlocks.
This article explains how to build a large‑scale, real‑time vehicle monitoring system using Apache Storm and Kafka on Alibaba Cloud, covering the challenges of big‑data ingestion, system architecture, deployment steps, performance testing, and practical lessons learned.
This article provides a comprehensive overview of Apache Spark, covering its origins, core concepts such as RDDs, transformations, actions, dependencies, execution modes, and key components like Spark SQL, Streaming, MLlib, and GraphX, while also offering practical code examples and visual illustrations.
Since 2011, LinkedIn has expanded its Kafka deployment from handling billions to over a trillion messages per day, focusing on quotas, a new ZooKeeper‑free consumer, reliability enhancements, security, monitoring frameworks, fault‑injection testing, cluster balancing, and ecosystem integrations, offering valuable lessons for large‑scale streaming systems.
Spark’s big‑data analytics ecosystem comprises core components such as the in‑memory RDD data structure, Streaming for real‑time processing, GraphX for graph analytics, MLlib for machine‑learning, Spark SQL for querying, the Tachyon file system, and SparkR, each enabling scalable, distributed computation.
This article provides a comprehensive overview of Kafka, covering its background, design goals, architecture, key terminology, message routing, consumer groups, delivery guarantees, and a comparison with other popular message queue systems such as RabbitMQ, Redis, ZeroMQ, and ActiveMQ.
