Tagged articles
real-time streaming
100 articles · Page 1 of 1
The article details how a live‑streaming data team built an AI‑assisted, metric‑driven system that lets users describe a real‑time data task in under 200 characters, automatically generates DSL and Flink SQL, and supports safe incremental adjustments via a Hook protocol, cutting development time from days to minutes.
AI-assisted data pipelineDSLFlink
0 likes · 17 min read

The article details how Taotian Group’s senior data engineer Zhu Ao designed a lake‑stream integrated architecture using Fluss for second‑level real‑time storage, Paimon for minute‑level lakehouse persistence, and StarRocks as a unified OLAP query layer, achieving over 50% faster development and more than 80% cost reduction.
FlussOLAPPaimon
0 likes · 14 min read


Niu LiuJul 13, 2026 · Artificial Intelligence
The article walks through constructing a full‑stack real‑time recommendation system—from user‑behavior collection via Kafka, through Flink streaming jobs for hot‑list, user and item profiling, to storage in Redis, HBase and Elasticsearch, and finally a React/Ant Design console that visualizes the pipeline and enables debugging.
ElasticsearchFlinkHBase
0 likes · 12 min read

The article outlines practical engineering steps to build a sub‑200 ms end‑to‑end real‑time data pipeline, covering latency budgeting, data ingestion with gRPC/Kafka, stream engine selection (Flink 2.x, RisingWave), state backend tuning, output sink choices, and monitoring with backpressure and OpenTelemetry.
Apache FlinkKafkaOpenTelemetry
0 likes · 13 min read

ShotStream tackles the high latency and zero‑interaction problems of multi‑shot long video generation by proposing a streaming architecture with a dual‑cache memory, discontinuous RoPE, and a two‑stage self‑forcing distillation, achieving over 25× speedup to 16 FPS on a single H200 GPU and outperforming existing bidirectional and autoregressive models.
ECCV 2026ShotStreamdual cache
0 likes · 8 min read

Alibaba Cloud Message Queue for Kafka introduces a native message‑to‑lake capability that integrates Apache Iceberg with OSS Table Bucket, eliminating Spark/Flink/Kafka Connect, providing exactly‑once semantics, automatic schema management, dual write modes, smart partitioning, and up to ten‑fold performance gains across diverse real‑time analytics scenarios.
Apache IcebergKafkaOSS Table Bucket
0 likes · 12 min read

The article presents a Flink‑Fluss‑LLM architecture that captures full‑link agent events via a non‑intrusive hook, combines semantic AI inference with deterministic CEP rules, and delivers millisecond‑level alerts for malicious user detection, tool result poisoning, and chain‑attack risk mitigation.
AI FunctionFlinkFluss
0 likes · 41 min read

The article details how JD’s JoyStreamer and JoyStreamer‑Flash models overcome text‑control weakness, multimodal conflict, and identity drift to enable long‑duration, free‑state, real‑time interactive digital‑human video generation, surpassing current SOTA models in benchmark scores and reaching 30 FPS inference speed for e‑commerce live streaming.
Digital HumanGenerative AIJoyStreamer
0 likes · 12 min read

The 2025 release of Apache Spark 4.0 brings a comprehensive overhaul—including default ANSI SQL mode, full SQL scripting support, a new Real‑Time streaming mode, adaptive query execution, dynamic memory management, and GPU‑accelerated MLlib—significantly boosting performance, reliability, and developer productivity across big‑data workloads.
Apache SparkGPU accelerationSQL
0 likes · 9 min read

This article compares Spring Boot 3's StreamingResponseBody and SseEmitter approaches for server‑side streaming, providing complete backend and frontend implementations, code samples, and performance considerations to help developers choose the right solution for real‑time data delivery.
Java backendServer-Sent EventsSseEmitter
0 likes · 8 min read

Learn how to transform PHP logs into real‑time, structured events by implementing a log event pipeline that includes JSON logging, lightweight collectors like Filebeat, streaming platforms such as Kafka or Flink, enrichment, and visualization with Grafana, enabling instant monitoring, alerting, and data‑driven decisions.
FlinkGrafanaKafka
0 likes · 7 min read

This article details the redesign of a code coverage platform, introducing a Kafka‑centric real‑time data stream, language‑agnostic contracts, non‑intrusive agents for Java and PHP, fine‑grained multi‑dimensional analysis, and interactive visualizations that dramatically improve testing efficiency and coverage accuracy.
Multi-languageagent architecturecode coverage
0 likes · 11 min read

This article explains how Alibaba Cloud’s big‑data platform leverages Flink ML to build an intelligent‑operations service that tackles stability, cost and efficiency challenges through time‑series anomaly detection, forecasting and streaming log‑clustering, dramatically reducing latency, complexity and operational overhead.
FlinkIntelligent Operationslog clustering
0 likes · 25 min read

Server‑Sent Events (SSE) offer a lightweight, HTTP‑native solution for real‑time, one‑way streaming, making them ideal for AI chat applications that need a typewriter‑style response, with advantages over WebSockets such as easy implementation, automatic reconnection, broad compatibility, and simple debugging.
JavaScriptPythonSSE
0 likes · 13 min read

Learn how to implement efficient real-time log streaming in Spring Boot using ResponseBodyEmitter, covering its purpose, core methods, usage scenarios, step-by-step controller code, connection lifecycle management, comparison with SSE and Streaming, and important considerations for client support, timeouts, and thread safety.
HTTP Chunked TransferJavaResponseBodyEmitter
0 likes · 11 min read

This article introduces Spring Boot 3’s StreamingResponseBody, explains its benefits for high‑concurrency scenarios, and provides three hands‑on examples—large file download, real‑time data streaming, and dynamic CSV export—complete with full Java code and practical tips for efficient backend development.
CSV exportFile DownloadSpring Boot
0 likes · 8 min read

The article describes a Flink‑based real‑time anti‑fraud streaming system that combines a risk‑control platform, configurable YAML‑driven pipelines, and optimized state handling—using early event‑time triggers, micro‑batch caching, and coarse‑grained key reduction—to compute multi‑dimensional features, support rapid strategy updates, simulation filtering, and seamless output to ClickHouse, Hive, and Redis for both instant monitoring and offline analysis.
Anti-FraudConfigurationFlink
0 likes · 26 min read

The article details a Flink‑based real‑time anti‑cheat streaming architecture that combines tumbling, sliding and session windows with early triggers, batch state updates cached in memory, coarse‑grained key reduction, and YAML‑driven strategy configuration to deliver millisecond‑level detection, seamless integration with ClickHouse, Hive, Redis and message queues, and self‑service analytics, achieving high throughput, low latency, and robust stability for large‑scale risk control.
Configuration ManagementFlinkanti-cheat
0 likes · 25 min read

This article explains how Server‑Sent Events (SSE) work, outlines the limitations of the native EventSource API, and demonstrates how the fetch‑event‑source library enhances SSE with POST support, custom headers, retry strategies, and visibility handling, enabling efficient real‑time AI data streams in modern web front‑ends.
AIJavaScriptSSE
0 likes · 6 min read

This guide explains the concept of stream output, its advantages and typical scenarios, then provides step‑by‑step implementations of Server‑Sent Events and HTTP chunk streaming using the Webman framework, including PHP process code, configuration, startup commands, and front‑end testing.
HTTP ChunkPHPSSE
0 likes · 9 min read

This article presents an in‑depth analysis of Kafka's native throttling mechanisms, identifies their limitations in large‑scale e‑commerce scenarios, and introduces JDQ's multi‑dimensional, dynamic throttling architecture that ensures stable throughput and priority‑aware bandwidth management across broker failures and traffic spikes.
JDQKafkabandwidth management
0 likes · 17 min read

This guide walks through the challenges of ingesting high‑volume sales‑point data into ClickHouse, explains why a buffering layer like Kafka is essential, compares three integration approaches—ClickHouse Kafka engine, Kafka Connect, and an external writer—and provides concrete SQL and Terraform examples to help you choose the right solution.
ClickHouseExternal writerKafka
0 likes · 13 min read

The article details Goldsky's multi‑tenant architecture that combines Redpanda, Apache Flink, and ClickHouse to stream TiB‑scale blockchain data in real time, explains the design choices, challenges, and lessons learned, and showcases a public ClickHouse dataset for the Base blockchain.
Apache FlinkBlockchain DataClickHouse
0 likes · 14 min read

This article introduces Baidu's log platform that handles billions of daily events, explains UBC logging concepts and monitoring requirements, and details a low‑cost, high‑accuracy architecture using real‑time streaming, dimension mapping, watermarking, and time‑window aggregation to achieve reliable, scalable event monitoring.
UBCWatermarkbig data
0 likes · 14 min read

This article explains Baidu's log‑center architecture for handling billions of UBC events per day, detailing UBC concepts, monitoring requirements, a low‑cost scalable design with dimension mapping, watermarking, data trimming and time‑window aggregation, and the resulting performance and cost benefits.
Cost OptimizationUBCWatermark
0 likes · 13 min read

RiskFactor unifies iQIYI’s legacy real‑time and offline feature platforms onto Opal’s DAG‑plus‑SQL engine, accelerating feature production fifteen‑fold, cutting latency from hours to minutes, streamlining development, lowering costs, and delivering more reliable, versioned risk‑control capabilities against sophisticated online threats.
DAGRisk Controlbig data
0 likes · 14 min read

This tutorial explains how to set up a real‑time camera monitoring system using PHP, covering hardware connection, required streaming server software, a complete PHP script to fetch and stream video, and instructions for running and security considerations for local environments.
CameraPHPVideo Monitoring
0 likes · 4 min read

This guide walks through setting up Flink CDC with MySQL on SpringBoot 2.7, covering binlog configuration, Maven dependencies, Java implementation for real‑time change capture, startup options, a custom Redis sink, and a web UI for monitoring the streaming pipeline.
CDCFlinkJava
0 likes · 10 min read

This article analyzes the challenges of commercial real‑time data processing—such as stability, multi‑stage computation, and frequent schema changes—and explains how the RTS platform provides end‑to‑end managed solutions, auto schema handling, primary‑secondary redundancy, experiment‑first deployment, and metadata generation to unlock high‑velocity data value for advertising operations.
RTS platformSchema Managementadvertising analytics
0 likes · 17 min read

This article examines the trade‑offs of using Flink versus Doris/StarRocks for real‑time data pipelines, highlighting Flink's strengths and pain points, and proposes shifting computation to the OLAP layer with Doris to reduce development and operational costs while maintaining near‑real‑time performance.
DorisFlinkOLAP
0 likes · 5 min read

The article chronicles Netflix’s evolution from a failing batch pipeline to a cloud‑native, self‑service streaming platform, detailing four development phases, the technical challenges faced, the stream‑processing patterns introduced, key learnings, and future opportunities for real‑time data and machine‑learning workloads.
Data PlatformFlinkKafka
0 likes · 30 min read

Explore the fundamentals of cloud gaming and discover how WebRTC's real-time audio‑video, NAT traversal, and data channel technologies enable low‑latency game streaming across devices, with detailed architecture, protocols, and code examples for developers.
NAT traversalWebRTCcloud gaming
0 likes · 36 min read

Bilibili’s Cloud Multimedia team uses AI‑driven pipelines to cut bandwidth costs while delivering low‑bitrate, high‑quality video, employing a QoE‑based decision engine, real‑time 4K super‑resolution for game streams, low‑rank reconstruction for narrow‑band HD, data‑driven HDR LUTs, and explores diffusion‑based restoration for legacy content.
AIBilibililow bitrate
0 likes · 27 min read

This article explains how Tencent builds a trillion‑scale real‑time signal system for its content ecosystem, covering signal applications, data source and processing challenges, a layered architecture with Flink‑based streaming, dynamic topic detection, high‑throughput ID mapping, large‑window calculations, rule‑engine triggering, and future roadmap for scalability and cost reduction.
FlinkTencentreal-time streaming
0 likes · 17 min read

This article details Autohome's transition from an hourly offline ad data warehouse to a Flink‑StarRocks real‑time architecture, covering background, engine and storage selection, multi‑layer design, implementation steps, encountered issues, monitoring strategies, and future roadmap to achieve second‑level data freshness and high accuracy.
AdvertisingFlinkStarRocks
0 likes · 12 min read

The article explains how market trends, data‑driven enterprise needs, and internal platform experience led Didi to build Know Streaming—a zero‑intrusion, plugin‑based real‑time streaming solution built on Kafka—to address scalability, operability, and community adoption challenges.
Data PlatformKafkaKnowStreaming
0 likes · 12 min read

Dolphin Streaming provides Alibaba’s advertising merchants with a DB‑like, SQL‑driven real‑time data platform built on Flink that abstracts storage and compute, enabling non‑engineers to develop, debug, and deploy streaming feature jobs quickly, boosting query volume, QPS, and revenue.
Data EngineeringDolphin StreamingFlink
0 likes · 13 min read


ITPUBDec 21, 2022 · Big Data
This article details Bilibili's extensive enhancements to the Flink runtime—including checkpoint recoverability, max‑parallelism calculations, State Processor API extensions, Full and Regional Checkpoints, hybrid HA, task‑level recovery, load‑balanced partitioners, and large‑scale cluster maintenance—to improve reliability and performance of its billion‑scale streaming workloads.
FlinkHigh Availabilitybig data
0 likes · 33 min read

The 2022 Flink Forward Asia summit showcased Apache Flink’s rapid community growth, key technical breakthroughs such as distributed snapshot upgrades, cloud‑native state storage, hybrid shuffle, Flink CDC 2.0, and Flink ML 2.0, and real‑world deployments at companies like Midea, miHoYo and Disney.
Apache FlinkFlink Forward AsiaStreaming Data Warehouse
0 likes · 25 min read

The 2022 Flink Forward Asia conference highlighted Apache Flink’s rapid growth, showcased major technical advances such as upgraded checkpointing, cloud‑native state storage, Hybrid Shuffle, Flink CDC 2.0, and Flink ML 2.0, and presented real‑world deployments from Alibaba, Midea, miHoYo, and Disney.
Apache Flinkdata integrationreal-time streaming
0 likes · 25 min read

The article explains how Blink, Alibaba’s optimized Flink variant, uses dynamic tables and streaming‑SQL to ingest mobile telemetry via source tables, compute per‑minute metrics such as API success rates with tumbling windows, and write results to Alibaba Cloud Log Service, enabling real‑time dashboards and extensible use cases like fraud detection.
FlinkSQLblink
0 likes · 10 min read

The article surveys the evolution of live streaming toward low‑latency, interactive scenarios and details WebRTC‑based real‑time audio/video system design, covering RTP/UDP transport, FEC and ARQ loss recovery, congestion control, jitter buffering, echo cancellation, edge‑node path optimization, and a multi‑layer architecture with signaling, routing, mixing services for scalable, high‑availability PK deployments.
Media ServerWebRTCaudio-video
0 likes · 16 min read

Lancer, Bilibili’s real‑time streaming backbone, has evolved from a monolithic Flume pipeline to a log‑id‑isolated, Kubernetes‑native architecture where Go edge agents feed synchronous Kafka‑proxied gateways into per‑logid topics processed by dedicated Flink‑SQL jobs, delivering exactly‑once, back‑pressured, highly scalable data ingestion for billions of daily requests.
FlinkKafkaLancer
0 likes · 29 min read

This article details Kuaishou’s real‑time data warehouse architecture and its comprehensive assurance framework—including forward lifecycle standards, reverse fault‑injection testing, and Spring Festival event practices—highlighting challenges of massive traffic, high timeliness, accuracy, and stability, and outlining future plans for automation, batch‑stream integration, and cost reduction.
FlinkSLAdata warehouse
0 likes · 23 min read

ZRTC, the real‑time audio‑video platform behind 作业帮, has been refined for over three years to support massive, multi‑cloud, multi‑protocol live streaming, employing a unified SDK, intelligent scheduling, custom SFU services, and extensive performance tuning to achieve high concurrency, low latency, and robust high‑availability.
WebRTCcloud-nativehigh concurrency
0 likes · 22 min read

This article explains Baidu's evolving inspection scheduling system for its smart mini‑programs, detailing the challenges of massive page volumes, the V1.0 offline architecture, the V2.0 real‑time enhancements, resource constraints, deduplication logic, and the measurable improvements in risk detection and ecosystem health.
big datacloud computingcontent safety
0 likes · 17 min read

The article outlines five major 2022 data trends— the rise of analytics engineers, the intensifying lake‑house competition, the growth of real‑time streaming pipelines and operational analytics, the expanding cloud marketplaces for data tools, and the push toward unified data‑quality terminology—explaining their origins, market impact, and future outlook.
Data Engineeringanalytics engineeringcloud marketplace
0 likes · 21 min read

This article presents a step‑by‑step tutorial on using the OpenMLDB Pulsar Connector to stream real‑time data from Apache Pulsar into OpenMLDB, covering connector architecture, key features, Docker‑based installation, sink configuration, schema registration, message production, verification queries, and future roadmap details.
Apache PulsarConnectorOpenMLDB
0 likes · 13 min read

JD Cloud’s new mobile super‑resolution SDK leverages deep‑learning ESPCN algorithms with ROI‑based processing to upscale video streams in real time, delivering up to 80% longer playback, 30% lower bandwidth costs, and measurable quality gains demonstrated through PSNR, VMAF, and SSIM metrics.
Bandwidth ReductionESPCNdeep learning
0 likes · 6 min read

This guide details an end‑to‑end breast‑cancer prediction pipeline on Tencent Cloud, covering offline decision‑tree training with TI‑ONE, model packaging as a PMML service, real‑time feature generation via Oceanus and CKafka, and live inference stored in ClickHouse, all within a secure VPC.
AIFlinkTencent Cloud
0 likes · 19 min read

The article describes the background, challenges, architecture, operator design, DAG processing, tag persistence, and business applications of a real-time dynamic tag processing platform (CDP) built to improve revenue growth and cost reduction for Trip.com's international operations.
DAGTag ManagementTiDB
0 likes · 16 min read

Learn how to capture, stream, and visualize real-time IMU sensor data using MQTT and Grafana Live’s streaming API, with step-by-step setup on a Raspberry Pi, configuration of the Grafana MQTT datasource plugin, and visualization via time-series and timeline panels.
Raspberry Pireal-time streamingsensor data
0 likes · 7 min read

BIGO replaced its Kafka‑based message‑flow platform with Apache Pulsar to overcome scaling, stability, and operational cost challenges, leveraging Pulsar’s storage‑compute separation, seamless horizontal expansion, low latency, and tight integration with Flink for real‑time ETL and AB‑test pipelines, resulting in billions of messages processed daily with half the hardware cost.
Apache PulsarETLFlink
0 likes · 17 min read

This article details JD's end‑to‑end design of a real‑time browsing record platform that captures, stores, and queries up to 200 recent items per user with millisecond latency, covering architecture, hot‑cold data separation, microservice APIs, and streaming pipelines using Kafka, Flink, Jimdb, and HBase.
FlinkHBaseJimdb
0 likes · 13 min read

The Xianyu “Play” tab flow‑control system combines task‑specific and rule‑based strategies with a dynamic strategy‑, control‑, and distribution‑chain architecture that integrates real‑time data processing into the recommendation engine, delivering guaranteed exposure, boosting daily posts by 14.4 % and paving the way for multi‑objective, zero‑code control.
Flow Controlbig datacontent recommendation
0 likes · 6 min read

Beike's real-time computing platform, led by Liu Liyun, has evolved from early Spark Streaming to a Flink-based system with SQL 1.0, 2.0, and upcoming 3.0, supporting a large-scale data warehouse, event-driven processing, extensive monitoring, and diverse business scenarios across the company's operations.
Flinkdata warehouseevent-driven
0 likes · 14 min read

iQIYI transformed its data infrastructure from a traditional offline T+1 model to a comprehensive real‑time ecosystem—leveraging Kafka, Flink, a three‑layer Stream Data Service Platform, the Talos drag‑and‑drop pipeline, and a Druid‑based analytics platform—to enable low‑latency monitoring, personalized recommendations, ad targeting, and continuous machine‑learning workflows while planning future stream‑batch integration and lake‑warehouse convergence.
AnalyticsFlinkKafka
0 likes · 13 min read

This article presents NetEase Cloud Music's real‑time computing platform built on Flink and Kafka, covering background, architectural design, Kafka and Flink selection reasons, platformization, warehouse usage, encountered challenges, and the solutions implemented to improve reliability and performance.
FlinkKafkaPlatform Design
0 likes · 11 min read

Zheng Zhisheng from Bilibili presents the Saber real-time computing platform, detailing its pain points, evolution, Apache Flink‑based architecture, SQL‑centric BSQL programming, DAG drag‑and‑drop design, AI use cases, and future development plans to improve scalability, operability, and AI integration.
AI IntegrationApache FlinkBSQL
0 likes · 19 min read

This article describes the design and implementation of an intelligent traffic distribution system for a reading platform, detailing its background, overall architecture, sub-modules such as the small‑traffic experiment platform, near‑line computation, retrieval strategies, pacing algorithms, and how it balances user personalization with content ecosystem growth.
AIPersonalizationRecommendation System
0 likes · 8 min read

This article explains how VIPKID leverages real‑time streaming with Flink to build a unified data platform, automatically tag and process help requests during 1‑v‑1 live classes, and achieve significant reductions in manual monitoring while improving course quality and user experience.
Flinkautomationbig data
0 likes · 14 min read

The article explains how a flexible, high‑performance user‑tagging system is built on a batch‑stream integrated architecture using big‑data technologies such as Impala, HDFS, and Flink to support both offline and real‑time label generation for precise marketing, product improvement, and operational analytics.
FlinkImpalaUser Tagging
0 likes · 15 min read

This article details Kuaishou's Flink‑based real‑time computing architecture, its massive cluster scale, and the comprehensive strategies—including overload protection, system stability, pressure testing, and resource guarantees—implemented to ensure reliable streaming for the 2020 Spring Festival Gala and its real‑time dashboard.
FlinkKuaishouSpring Festival Gala
0 likes · 12 min read

The article details Beike's Data Management Platform (DMP) built since May 2018, covering its overall architecture, data collection, processing, real-time profiling, storage solutions, application scenarios, achieved performance metrics, and future development directions.
BeikeDMPData Engineering
0 likes · 9 min read

Facing billions of video entities and edges, Youku’s engineering team replaced traditional relational stores with a graph‑based knowledge platform, leveraging Alibaba’s Blink streaming engine and Lindorm to enable real‑time, incremental updates, unified UDF logic, and scalable feature computation for search and recommendation.
big datagraph databaseknowledge graph
0 likes · 10 min read

This article explains Flink's role in HuoLaLa's risk‑control system, covering its background, the Lambda‑style architecture that combines batch and streaming, the real‑time data pipeline, machine‑learning models, and operational safeguards that together enable proactive fraud detection.
FlinkLambda architecturebig data architecture
0 likes · 16 min read

This article presents the evolution, architecture, and operational metrics of Beike Zhaofang's Hermes real‑time computing platform built on Apache Flink, detailing its business scale, SQL editors, task growth, monitoring, use cases, and future development directions.
Apache FlinkData EngineeringSQL
0 likes · 10 min read

This article presents a comprehensive real‑time and offline integrated solution for a channel analysis system, detailing challenges, architecture, implementation using Flink, Spark Streaming, Kafka, Elasticsearch, and HIVE, and demonstrating minute‑level latency and high accuracy through performance evaluations.
ElasticsearchFlinkKafka
0 likes · 10 min read

The article details how 58.com designed and evolved its one‑stop real‑time computation platform Wstream, migrating from Storm and Spark Streaming to Apache Flink, and describes the architecture, task isolation, stream‑SQL features, monitoring, and ongoing optimizations that enable processing of over 600 billion records daily.
FlinkStream SQLTask Migration
0 likes · 12 min read

This article explains how to dynamically update configuration data in real‑time Spark Streaming and Flink jobs using broadcast variables and broadcast state, providing Java code examples and discussing the limitations and practical considerations of each approach.
BroadcastFlinkSpark
0 likes · 8 min read


21CTONov 27, 2019 · Big Data
The article summarizes Guo Yi’s 2019 Alibaba Cloud conference talk, outlining Xiaohongshu’s personalized recommendation architecture, detailing the data stack from ingestion to warehouse, and showcasing a Flink‑based real‑time multi‑dimensional user behavior aggregation use case, followed by a vision for the next year’s data architecture evolution.
FlinkPersonalizationdata architecture
0 likes · 3 min read

The article details the technical evolution of 58.com’s real-time computing platform—from Storm and Spark Streaming to a Flink‑based one‑stop solution called Wstream—covering use cases, architecture, stability measures, migration from Storm, operational diagnostics, and future development plans.
FlinkStorm MigrationWstream
0 likes · 11 min read

This article details the technical evolution of 58.com’s real‑time computing platform, describing the shift from Storm and Spark Streaming to Apache Flink, the design of the one‑stop Wstream platform, its large‑scale deployment, stability measures, SQL streaming capabilities, task migration, diagnostics, optimizations, and future plans.
FlinkTask MigrationWstream
0 likes · 11 min read

This article details how Weibo’s advertising team designed and implemented a real‑time data platform capable of processing over a hundred billion daily logs, covering technology selection, Flink advantages, architecture evolution, data processing pipelines, component libraries, fault‑tolerance strategies, and the construction of a multi‑layer real‑time data warehouse.
Flinkbig datacheckpoint
0 likes · 25 min read

This article introduces Alibaba's Canal project, explaining its background, architecture, and how it parses MySQL binlog for real‑time data subscription, and provides a complete Java client example with code snippets to demonstrate connecting, fetching, and processing binlog events.
CanalJavaMySQL
0 likes · 9 min read

This article describes how Autohome migrated its real‑time analytics from Storm to a Flink‑SQL platform, detailing the architectural design, development and operational advantages, practical use cases such as recommendation metrics, and future plans for ecosystem expansion and open‑source release.
FlinkSQLdata warehouse
0 likes · 12 min read

The article details the six‑layer big‑data architecture of an online school, chronicles its migration from Storm to Spark Streaming and finally to Flink, and showcases concrete real‑time applications such as gateway monitoring, user‑profile tagging, renewal reporting, and advertising analysis, while outlining future development directions.
AnalyticsFlinkSpark Streaming
0 likes · 14 min read

This article recounts Tencent's journey from adopting Flink to building the Oceanus platform, detailing its architecture, product features, and a series of deep extensions—including UI redesign, JobManager failover, checkpoint handling, enhanced windows, LocalKeyBy, watermark idle detection, and log isolation—aimed at supporting trillion‑scale real‑time data processing.
FlinkOceanusTencent
0 likes · 18 min read

The paper presents a high‑performance real‑time data processing pipeline that collects, transports, preprocesses, and computes service logs and metrics using Alibaba Logtail, LogHub, and an enhanced Flink (Blink) engine, persisting root‑cause graphs in Lindorm, achieving sub‑3‑second latency for tens of millions of events per second and cutting diagnosis time to about five seconds.
Flinkarchitecturedata processing
0 likes · 10 min read

The article presents Tencent's evolution of real‑time stream processing using Flink, the design of the Oceanus one‑stop visual platform, and a series of deep extensions and optimizations—including UI redesign, JobManager failover, checkpoint handling, enhanced windows, LocalKeyBy, idle detection, and log isolation—aimed at supporting petabyte‑scale data workloads.
FlinkOceanusTencent
0 likes · 16 min read

This article explains how Alibaba's Blink engine tackles the complex challenges of building a real‑time supply‑chain data warehouse—covering retroduction, dimension‑table joins, data skew, timeout statistics, zero‑point optimizations, and future directions—through SQL‑based stream processing and intelligent resource tuning.
Data SkewDimension JoinFlink
0 likes · 14 min read

This article summarizes Wang Xinchun's 2018 DAMS China Data Asset Management Summit talk, detailing the current state, core services, responsibilities, evolution, architecture, challenges, and future directions of a large‑scale real‑time data platform built on Storm, Spark, Flink, and Kubernetes, including a unified data management approach.
Data PlatformFlinkKubernetes
0 likes · 22 min read

This article walks through Youzan's real‑time platform architecture, explains why Flink was chosen over Spark Structured Streaming, details practical challenges such as container over‑provisioning and monitoring overhead, shares solutions for Spring integration and async caching, and outlines future directions for SQL‑based streaming and scheduler improvements.
FlinkSQLYARN
0 likes · 19 min read

This article outlines Didi's journey from fragmented, self‑built real‑time clusters to a unified, YARN‑managed platform that leverages Spark Streaming, Flink, and StreamSQL, detailing architectural choices, resource isolation, CEP enhancements, and the resulting impact on latency‑critical services.
CEPFlinkSpark Streaming
0 likes · 10 min read

This article describes how a large‑scale operations monitoring system at Ctrip evolved from rule‑based alerts to a deep‑learning‑driven anomaly detection pipeline using LSTM models and Flink streaming, achieving a ten‑fold reduction in alert volume while improving fault detection.
FlinkLSTManomaly detection
0 likes · 14 min read

To meet Double‑Eleven’s sub‑second, billion‑item feed demands, Alibaba’s Xianyu selection system evolved from a Solr‑based search pipeline through offline batch and PostgreSQL attempts to a Blink‑powered real‑time stream platform using Niagara’s low‑latency LSM storage, delivering high‑throughput, personalized product feeds.
AlibabaFlinkXianyu
0 likes · 23 min read


21CTOOct 19, 2018 · Big Data
This article summarizes Meituan’s real‑time computing platform, detailing its layered architecture built on Kafka, Flink on YARN, state management, resource isolation, fault tolerance, monitoring, and the Petra metric aggregation system, while highlighting the challenges faced and the solutions implemented to achieve high‑throughput, low‑latency stream processing at massive scale.
FlinkState Managementbig data
0 likes · 18 min read


21CTOApr 28, 2018 · Big Data
This article explains why Kafka has become essential for real‑time data streaming in the big‑data era, detailing its performance advantages, core use cases, major adopters, multilingual support, and how its scalable storage and retention mechanisms empower modern data pipelines.
Kafkareal-time streaming
0 likes · 10 min read

Ctrip’s real‑time user behavior service, a core foundation for recommendations, ads, and user profiling, was redesigned with a Java‑based stack (Kafka, Storm, Redis, MySQL) to achieve millisecond‑level latency, high availability, and ten‑fold scalability across processing and output flows.
KafkaMySQLRedis
0 likes · 12 min read


21CTOFeb 20, 2018 · Big Data
This article explains how real-time streaming has evolved from batch Hadoop systems through Lambda architecture to modern Kappa-style pipelines, highlighting its growing importance for developers, enterprises, and the integration of streaming with microservices, AI, and cloud-native technologies.
AI IntegrationKappa architectureLambda architecture
0 likes · 8 min read

The article details WMDA, a no‑code and manual‑code data collection platform for PC, mobile and app that supports real‑time and offline user behavior analysis, describing its functional model, behavior taxonomy, five‑layer architecture, tracking techniques, circle‑selection, data services, streaming and batch processing pipelines, and related technologies such as Storm, Spark, Druid and Roaring Bitmap.
DruidRoaring Bitmapbig data
0 likes · 18 min read


ITPUBNov 13, 2017 · Big Data
The article explains how continuous data generation and real‑time stream processing enable e‑commerce platforms like NetEase Kaola to handle massive Double 11 traffic, showcasing use cases, architectural shifts from batch to incremental computing, and the technical challenges of latency, accuracy, and fault tolerance.
SQLdistributed systemse-commerce
0 likes · 15 min read


21CTOAug 23, 2017 · Big Data
This article explains the design of a real-time customer behavior collection platform that uses NSQ for messaging, Storm for streaming processing, and HBase for storage, covering architecture, data flow, reliability guarantees, and deployment considerations.
HBaseNSQStorm
0 likes · 11 min read

Alibaba algorithm engineer Shi Xiaogang shares his journey from a Peking University PhD researching real‑time iterative computation on data streams to developing Blink’s state management and recovery features in Flink, highlighting the challenges of transitioning from academia to industry and the impact of large‑scale real‑time systems.
Algorithm EngineeringFlinkcareer interview
0 likes · 13 min read

Alibaba’s Double 11 live‑data dashboards required ultra‑high‑precision, low‑latency real‑time processing of billions of events, and the article explains the end‑to‑end architecture—including DRC, TimeTunnel, Galaxy, OTS, XTool, and OneService—used to achieve million‑plus QPS, fault‑tolerance, and flexible data collection.
AlibabaTimeTunnelbig data architecture
0 likes · 14 min read

This article explains the design and implementation of the DWS platform, detailing its log‑driven architecture with Dbus, Wormhole, and Swifts, the technical choices behind real‑time data extraction, transformation, and delivery, and real‑world use cases in finance.
CDCCanalKafka
0 likes · 22 min read

This article describes Ctrip's development of a unified real-time data platform, detailing its motivations, architectural choices such as Kafka and Storm, implementation of shared schemas, resource control, monitoring, and operational lessons, as well as experiences with Storm, JStorm, and Streaming CQL.
CtripData PlatformKafka
0 likes · 15 min read


21CTODec 21, 2015 · Fundamentals
With broadband speeds soaring, UDP’s simplicity and low latency now outperform TCP in web browsing, streaming, gaming, and IoT, as real‑world tests from Akamai and Google show dramatically lower packet loss, faster connections, and reduced buffering, signaling a shift toward UDP‑centric networking.
QUICTCPUDP
0 likes · 5 min read

The article details the progressive architectural decisions and technical solutions behind a real-time multi‑party audio/video streaming platform, covering format research, backend service design, protocol choices, codec selection, node topology redesign, and video integration to achieve low‑latency, cross‑platform communication.
HLSOpus codecWebRTC
0 likes · 15 min read

This article examines Baidu’s home‑grown real‑time big‑data platforms Dstream and TM, detailing their architectures, performance metrics, key features, and practical use cases such as log ETL and real‑time bidding, while highlighting how they meet millisecond‑level processing demands.
BaiduDstreamTM platform
0 likes · 9 min read
