real-time data warehouse

Bilibili Tech

Apr 8, 2025 · Big Data

Building a Real-Time Data Warehouse for B站 Game Business

To meet Bilibili’s rapidly expanding game business, the team built a unified real-time data warehouse using Hologres and Flink that replaces the traditional Lambda stack, delivering high-throughput writes, low-latency processing, seamless offline-online integration, global deployment, and real-time support for operations, advertising, and risk analytics.

Data architecture case studyFlinkGame business data

0 likes · 17 min read

Building a Real-Time Data Warehouse for B站 Game Business

iQIYI Technical Product Team

Mar 27, 2025 · Big Data

Cost‑Effective Real‑Time Data Warehouse 2.0: Migrating from Kafka to Iceberg

iQIYI transformed its real‑time data warehouse by replacing a costly Kafka‑based Lambda stack with a unified stream‑batch Iceberg lake, cutting storage expenses by 90%, halving compute costs, extending data retention, and delivering minute‑level freshness for 90% of use cases while preserving second‑level processing where needed.

Cost OptimizationFlinkIceberg

0 likes · 11 min read

Cost‑Effective Real‑Time Data Warehouse 2.0: Migrating from Kafka to Iceberg

Alimama Tech

Mar 12, 2025 · Big Data

Design and Evolution of Alibaba Advertising Real-Time Data Warehouse

Alibaba Mama’s advertising platform migrated from a monolithic Flink‑Kafka pipeline to a layered Paimon lakehouse, adding DWS upsert support and multi‑layer storage, which delivers minute‑level data freshness, cuts latency by 2.5 hours, reduces resource use over 40 %, halves development effort and achieves ≥99.9 % availability.

AlibabaFlinkPaimon

0 likes · 18 min read

Design and Evolution of Alibaba Advertising Real-Time Data Warehouse

DataFunSummit

Sep 20, 2024 · Databases

Key Topics and Abstracts from DataFun Summit: Graph DB, Vector DB, Real-Time Data Warehouses, and Cloud‑Native Solutions

The article presents a collection of technical abstracts from the DataFun Summit, covering XiaoHongShu's REDgraph distributed graph database, DingoDB's multimodal vector database, Tencent's Tianqiong autonomous data platform, real‑time data warehouse architectures at Douyin and Ant Group, and Alibaba Cloud's serverless ClickHouse offering, all aimed at advancing large‑scale data processing and analytics.

Big Datacloud nativegraph database

0 likes · 5 min read

Key Topics and Abstracts from DataFun Summit: Graph DB, Vector DB, Real-Time Data Warehouses, and Cloud‑Native Solutions

DataFunTalk

Sep 19, 2024 · Databases

Technical Topics Overview from DataFun Summit: Graph Database, Vector Database, Real-time Data Warehouse, and Cloud‑Native Solutions

The article presents a collection of technical overviews—including a graph database for distributed queries, a next‑generation vector database, real‑time data warehouse architectures at Douyin and Ant Group, a cloud‑native ClickHouse service, and best practices for financial data warehousing—while also explaining how to obtain the related e‑book.

Big DataData Engineeringcloud native

0 likes · 4 min read

DataFunSummit

Sep 18, 2024 · Big Data

Data Summit Abstracts: Graph Database, Vector Database, Real-time Data Warehouse, and Cloud‑Native Analytics

The article presents a series of technical abstracts covering Xiaohongshu's distributed graph database, DingoDB's multimodal vector store, Tianqiong's autonomous data‑warehouse innovations, Douyin's storage‑based real‑time warehouse, financial‑grade real‑time warehousing, Alibaba Cloud ClickHouse Serverless, best practices in financial data governance, and 58.com’s user‑profile warehouse construction.

Big Datacloud nativedata governance

0 likes · 5 min read

Data Summit Abstracts: Graph Database, Vector Database, Real-time Data Warehouse, and Cloud‑Native Analytics

DataFunTalk

Jul 25, 2024 · Big Data

Real‑time Data Warehouse Evolution with Data Lake: Challenges, Solutions, and Future Outlook

This article presents a comprehensive overview of JD Tech's real‑time data warehouse evolution, detailing the legacy Lambda architecture, its shortcomings, the integration of a data‑lake‑based solution, iterative redesigns, technical trade‑offs, and future directions for real‑time analytics.

Big DataClickHouseFlink

0 likes · 25 min read

Real‑time Data Warehouse Evolution with Data Lake: Challenges, Solutions, and Future Outlook

DataFunTalk

Jul 18, 2024 · Big Data

Ant Group's Real-Time Data Warehouse Architecture, Solutions, and Data Lake Outlook

This article presents Ant Group's recent exploration of real-time data warehouse architecture, covering its six-module design, data quality assurance mechanisms, stream‑batch unified processing with Flink and ODPS, and a forward‑looking data lake solution built on Paimon, offering practical insights for large‑scale streaming analytics.

Big DataFlinkdata lake

0 likes · 15 min read

Ant Group's Real-Time Data Warehouse Architecture, Solutions, and Data Lake Outlook

DataFunTalk

Jun 18, 2024 · Big Data

Real-time Data Warehouse Evolution with Data Lake: Architecture, Challenges, and Solutions

This article presents a comprehensive overview of the evolution from traditional Lambda‑based real‑time data warehouse solutions to a data‑lake‑integrated architecture, detailing the shortcomings of legacy designs, the iterative improvements made at JD Technology, and the technical and operational challenges encountered during implementation.

Big DataLambda architectureStreaming

0 likes · 24 min read

Real-time Data Warehouse Evolution with Data Lake: Architecture, Challenges, and Solutions

DataFunSummit

Apr 18, 2024 · Big Data

Real‑time Data Warehouse Evolution with Data Lake: Architecture, Challenges, and Solutions

This article presents a comprehensive overview of JD Tech's real‑time data warehouse evolution, detailing the legacy Lambda‑based design, its shortcomings, the transition to a data‑lake‑integrated architecture, iterative improvements, encountered technical and non‑technical issues, and future outlooks.

ClickHouseFlinkHudi

0 likes · 24 min read

DataFunSummit

Apr 8, 2024 · Big Data

Ant Group's Real-Time Data Warehouse Architecture, Solutions, and Data Lake Outlook

This article presents Ant Group's recent explorations and practices in real-time data warehousing, covering its modular architecture, data quality assurance mechanisms, stream‑batch integration techniques, graph‑based conversion attribution, and future data‑lake implementation using Paimon.

Big DataFlinkdata lake

0 likes · 15 min read

DataFunTalk

Mar 26, 2024 · Big Data

Building an Enterprise Real-Time Data Warehouse with Hologres and Flink at Cao Cao Mobility

This article presents a comprehensive case study of Cao Cao Mobility's transition from a traditional Lambda architecture to an enterprise‑grade real‑time data warehouse built on Hologres and Flink, detailing business background, pain points, architectural design, performance optimizations, metadata management, and future development directions.

Big DataData EngineeringFlink

0 likes · 20 min read

Building an Enterprise Real-Time Data Warehouse with Hologres and Flink at Cao Cao Mobility

DataFunTalk

Aug 21, 2023 · Databases

Case Study: Building a Real‑Time Log Data Analysis Platform with Apache Doris at China Unicom

This article describes how China Unicom’s Western Innovation Research Institute designed and deployed a centralized, real‑time log analytics platform using Apache Doris, detailing the migration from Hive and ClickHouse, performance optimizations, storage cost reductions, and the resulting improvements in data ingestion, query speed, and operational efficiency.

Apache DorisBig DataCold‑Hot Data Management

0 likes · 18 min read

Case Study: Building a Real‑Time Log Data Analysis Platform with Apache Doris at China Unicom

WeiLi Technology Team

Aug 2, 2023 · Big Data

How to Build a Real-Time Data Warehouse: Architectures, Challenges, and Industry Practices

This article examines the growing demand for real‑time data warehouses, compares mature streaming frameworks, evaluates Lambda, Kappa and hybrid architectures, reviews industry implementations from Didi and OPPO, and proposes a standard‑layer + stream + data‑lake solution with Apache Paimon, Hudi, and Iceberg.

Apache FlinkBig DataKappa architecture

0 likes · 27 min read

How to Build a Real-Time Data Warehouse: Architectures, Challenges, and Industry Practices

DataFunTalk

Jul 22, 2023 · Big Data

Optimization Practices for Real-Time Data Warehouse Governance at NetEase Cloud Music

This article details the current challenges, governance motivations, architectural design, and technical optimizations—including Flink SQL tuning, Kafka batch improvements, partitioned stream tables, containerization, and automated governance—implemented to enhance the efficiency, stability, and cost-effectiveness of NetEase Cloud Music's real-time data warehouse platform.

ContainerizationFlink optimizationKafka batch

0 likes · 23 min read

Optimization Practices for Real-Time Data Warehouse Governance at NetEase Cloud Music

DataFunSummit

May 24, 2023 · Big Data

NetEase Cloud Music Real-Time Data Warehouse Architecture and Low‑Code Platform Practices

This article presents a comprehensive overview of NetEase Cloud Music's real‑time data warehouse architecture, its stream‑batch consistency model, the low‑code FastX platform implementation, and future plans for expanding lake‑house capabilities and high‑availability solutions.

Big DataFlinkKafka

0 likes · 18 min read

NetEase Cloud Music Real-Time Data Warehouse Architecture and Low‑Code Platform Practices

DataFunTalk

May 17, 2023 · Databases

Evolution of 360 Commercial Real-Time Data Warehouse and Apache Doris Deployment

This article details the three‑stage evolution of 360's real‑time data warehouse—from Storm + Druid + MySQL to Flink + Druid + TiDB and finally to Flink + Apache Doris—explaining architectural pain points, the reasons for choosing Doris, and how the new system delivers sub‑second query latency, strong consistency, and simplified operations across advertising scenarios.

Apache DorisBig DataFlink

0 likes · 17 min read

Evolution of 360 Commercial Real-Time Data Warehouse and Apache Doris Deployment

DataFunTalk

May 5, 2023 · Big Data

NetEase Cloud Music Real-Time Data Warehouse Architecture and Low-Code Platform Practices

This article presents NetEase Cloud Music's real-time data warehouse architecture, covering its streaming and batch scenarios, layered design (ODS, CDM, ADS), technology stack choices, consistency mechanisms, the FastX low-code platform, and future development plans, offering a comprehensive technical overview for data engineers and architects.

Big DataClickHouseFlink

0 likes · 18 min read

DataFunSummit

Mar 19, 2023 · Databases

Building a Real-Time Unified Data Platform with Apache Doris: Insights from SelectDB

SelectDB shares its perspective on modern data analytics stacks, detailing the current challenges, the evolution of data architectures, and how Apache Doris enables a real‑time unified data foundation, while also reviewing Doris 1.2’s latest features, performance gains, and future roadmap.

Apache DorisBig DataData Analytics

0 likes · 18 min read

Building a Real-Time Unified Data Platform with Apache Doris: Insights from SelectDB

ByteDance Data Platform

Feb 15, 2023 · Databases

How ByteHouse Powers Real‑Time Data Warehousing at Scale

ByteHouse, a cloud‑native data warehouse built on ClickHouse, delivers ultra‑fast real‑time and massive offline analytics with elastic scaling, addressing business needs in ByteDance and the financial sector through optimized architecture, ROI‑driven monitoring, and comprehensive operational tools.

Big DataByteHouseClickHouse

0 likes · 16 min read

How ByteHouse Powers Real‑Time Data Warehousing at Scale