Tagged articles
Real-time Data Lake
13 articles · Page 1 of 1
This article details migrating from a legacy Canal+Kafka+Flink+Hive stack to a modern Flink CDC + Iceberg real-time data lake, covering Spring Boot orchestration, chunk-based full/incremental sync, Interval Join for wide tables, automated schema evolution, hidden partitioning, Exactly-Once checkpoint tuning, and Kubernetes deployment practices.
Apache IcebergFlink CDCKubernetes
0 likes · 17 min read

This article examines Apache Paimon’s innovative lakehouse architecture, detailing its LSM‑Tree storage, flexible merge engine, and multi‑engine integration, and showcases two real‑world deployments—an operator’s real‑time fraud‑prevention system and a manufacturing firm’s unified data platform—highlighting performance gains and cost reductions.
Apache PaimonReal-time Data Lakebig data
0 likes · 15 min read

Hisense Juhau, an AI‑enabled TV cloud service, overhauled its massive offline‑centric data platform by adopting a real‑time data lake, compute‑storage separation, and serverless Spark/StarRocks on Alibaba Cloud, achieving sub‑5‑minute data freshness, elastic scaling, and dramatically improved performance for personalized content recommendation and smart operations.
Apache PaimonReal-time Data Lakecloud computing
0 likes · 13 min read

This article explains how Apache Paimon can be used as a real‑time state store for Flink, detailing its low‑cost, scalable storage, lookup‑join design, table schema, bucket configuration, memory tuning, and practical use cases such as handling refund‑adjusted order tags and cumulative metrics.
Apache PaimonFlinkLookup Join
0 likes · 16 min read

The article reviews Apache Paimon's graduation to an Apache Top-Level Project, outlines the essential capabilities of modern lakehouse frameworks—including streaming and batch I/O, multi‑engine integration, and advanced features—and discusses the problems they solve and the promising direction of the lakehouse ecosystem.
Apache PaimonReal-time Data Lakebatch processing
0 likes · 5 min read

Apache Paimon, originally launched as Flink Table Store, has graduated to an Apache Top‑Level Project after a year of incubation, showcasing real‑time lakehouse capabilities, extensive ecosystem integration, and strong adoption by major enterprises, marking a significant milestone for streaming and batch data processing.
Apache PaimonReal-time Data Lakebig data
0 likes · 9 min read

This article shares Kangaroo Cloud's exploration and practice of a real-time data lake, covering background, data lake concepts, challenges, solution architecture using the Shuzhan platform with Iceberg/Hudi, CDC ingestion, small file handling, cross-cluster ingestion, materialized view acceleration, and future development plans.
CDCCross-Cluster IngestionHudi
0 likes · 12 min read

ByteDance’s data platform team explains their real‑time data lake implementation, covering its evolving definition, six core capabilities, challenges such as data management, concurrent updates, performance and log ingestion, and detailed case studies of multi‑stage deployment, indexing, metadata services, and future roadmap.
HudiReal-time Data Lakeindexing
0 likes · 32 min read

This article details Xiaohongshu's data platform engineering, describing how Apache Iceberg is leveraged for real‑time data lake ingestion, CDC pipelines, multi‑cloud storage, small‑file mitigation, schema evolution, and future plans across storage, compute, and management within a big‑data ecosystem.
Apache IcebergCDCFlink
0 likes · 16 min read

This article presents detailed case studies of ByteDance and Alibaba implementing real‑time data lake solutions with Hudi and Flink, describing the business drivers, architectural challenges, and the specific technical strategies such as unified metadata layers, optimistic locking, scalable hash indexing, and CDC‑based incremental ETL to achieve low‑latency, high‑throughput data processing.
FlinkHudiReal-time Data Lake
0 likes · 9 min read

ByteDance shares its real‑time data lake implementation, covering the evolving definition of data lakes, six core capabilities, challenges such as data management, weak concurrent updates, performance, and log ingestion, and detailed solutions including Hudi Metastore Server, bucket indexing, multi‑stage use cases, and future roadmap.
HudiReal-time Data Lakebatch processing
0 likes · 32 min read

This article explains how to integrate Apache Hudi with Flink CDC to build a near‑real‑time data lake, covering Hudi’s storage model, streaming primitives, version compatibility, Maven setup, SQL table definitions, data flow from MySQL through Kafka, and practical troubleshooting tips.
Apache HudiFlink CDCReal-time Data Lake
0 likes · 18 min read

This article details Baixin Bank's construction of a Flink‑driven real‑time computing platform integrated with Hudi as a real‑time data lake, covering background, architecture, data collection, transformation, storage layers, technical challenges, future roadmap, and practical lessons for similar big‑data initiatives.
Data EngineeringFlinkHudi
0 likes · 12 min read
