Tagged articles
real-time data processing
11 articles · Page 1 of 1
This article shares Kuaishou’s real‑time, massive‑scale model data processing pipeline, covering model scenarios, recommendation workflow complexity, large‑scale data storage, streaming joins, feature computation, NVM‑based storage solutions, strong consistency mechanisms, and future outlook for AI recommendation systems.
AIKuaishouNVM storage
0 likes · 16 min read

This article presents Kuaishou's end‑to‑end engineering solutions for handling massive, real‑time recommendation model data, covering scenario description, complex business pipelines, trillion‑parameter model storage, high‑throughput processing with Flink and NVM, and future directions for cloud‑native scalability.
KuaishouNVM storagelarge-scale models
0 likes · 15 min read

This article introduces Apache Kafka as a high‑throughput, scalable, fault‑tolerant distributed streaming platform, explains why it is chosen for real‑time data pipelines, and details key Kafka Streams concepts such as stream processing, interactive queries, stateful processing, windowing, serialization, and testing.
Apache KafkaKafka Streamsbig data
0 likes · 13 min read

The article explains how to archive Kafka real‑time data to Hive using either Flink SQL, which quickly creates partitioned ORC tables but requires timezone handling, or Flink DataStream for more complex pipelines, and offers best‑practice guidance on data quality, system complexity, security, and performance.
Data ArchivingDataStreamFlink
0 likes · 15 min read

Arctic, NetEase’s streaming lakehouse built on Apache Iceberg, unifies streaming and batch workloads with millisecond‑level latency, Hive compatibility, and built‑in message‑queue support, delivering CDC, upserts and OLAP without a Lambda architecture, as demonstrated by real‑time processing of 2 PB of Hive data for Cloud Music.
Apache IcebergArcticHive Compatibility
0 likes · 15 min read

The article describes how combining Apache Flink with Hudi enables an incremental data lake that delivers near‑real‑time analytics by switching to merge‑on‑read, fixing log handling bugs, improving compaction planning, and refactoring table‑service scheduling, while showcasing use cases such as CDC ingestion, data quality control, and real‑time materialized views, and outlines future enhancements like optimistic concurrency and unified schema evolution.
Apache HudiCDCCompaction Optimization
0 likes · 21 min read

This article details NetEase Yanxuan's evolution of a real‑time data platform from 2017 to present, covering background, current scale, layered architecture, Flink‑SQL development IDE, service‑oriented task execution, resource‑optimizing deployment modes, cloud‑native migration, comprehensive data governance, and future batch‑stream integration plans.
Flinkbig datacloud-native
0 likes · 15 min read

This article presents Car Home's experience building a real‑time materialized view system on Apache Flink, detailing system analysis, problem decomposition, a global‑version‑based CDC algorithm, its implementation as a Flink connector, practical deployment results, and remaining challenges such as clock dependency and state size.
AlgorithmCDCFlink
0 likes · 17 min read

This article explains how Alibaba built a comprehensive Blink testing platform—including code‑quality checks, functional, performance, stability, and pre‑release testing—to ensure the reliability and scalability of its real‑time big‑data processing engine during massive workloads like Double 11.
Apache FlinkTesting Frameworkbig data
0 likes · 13 min read

As mobile and IoT data surge, real-time stream computing—especially Flink’s low-latency, high-throughput, exactly-once engine—addresses challenges of latency, accuracy, and usability, and Tencent Cloud’s managed Flink service provides elastic, secure, integrated pipelines for applications ranging from online status monitoring to fraud detection and smart transportation.
Apache StormFlinkSpark Streaming
0 likes · 30 min read

This article describes how Ctrip migrated its large‑scale real‑time platform from JStorm to Spark Streaming, detailing the architectural design, the Muise Spark Core encapsulation, operational metrics, encountered pitfalls, and future plans to adopt Flink and Beam for streaming workloads.
Spark StreamingYARNbig data
0 likes · 22 min read
