Alibaba Mama evolved its advertising real-time data warehouse from a message queue-based architecture to a Flink + Paimon streaming lakehouse, solving structural defects like lack of deduplication, aggregation layer support, and schema enforcement by leveraging Paimon's primary key upsert and changelog capabilities, while sharing production optimization practices and comparing storage choices for advertising vs. risk control scenarios.
This article provides a practical decision framework for choosing between Apache Doris and StarRocks for real-time OLAP workloads, comparing their architectural origins, licensing, optimal use cases, and guiding teams through self-hosted versus managed deployment decisions based on organizational capacity and query profiles.
This article explores data quality challenges in real-time data warehouses, covering timing agreements for late-arriving data, reconciliation techniques using Flink and Paimon, SQL-based checking strategies, alerting with actionable details, and iterative quality validation processes.
Hologres evolves from a real-time data warehouse into an AI data infrastructure, integrating multimodal data via Object Tables and Dynamic Tables, providing AI Functions for hybrid search within SQL, an AI Assistant for automated warehouse operations, and a long-term memory service for cross-session agent recall.
This article details the end-to-end architecture of a real-time data warehouse, covering data ingestion via CDC and Kafka, stream processing challenges like deduplication and event-time handling, layered modeling (ODS/DWD/DWS/ADS), unified metric definitions, and critical production monitoring for latency, lag, data reconciliation, and consistency with offline results.
This article details how to build a production-ready real-time data warehouse using Flink, covering CDC ingestion, event-time processing, dimension joins, upsert aggregation, state TTL, checkpoint tuning, exactly-once verification, and a pre-launch checklist for reliability.
The article explains why real‑time data‑warehouse projects often fail in production and presents a complete, production‑ready solution that layers ODS, DWD, DWS and ADS using Flink CDC to capture MySQL changes, Kafka for buffering and replay, and Doris for OLAP storage, with detailed guidance on architecture, state handling, fault‑tolerance and operations.
Apache Paimon combines a lakehouse table format with an LSM-tree update engine to handle continuous updates, historical snapshots, and multi-engine access in real-time data warehouses, reducing the need for separate Kafka, Hive, and OLAP systems.
This article details a lightweight real-time data warehouse architecture using Apache Flink 2.0+ for streaming writes, Apache Paimon for lakehouse storage, Apache Doris for interactive queries, and a custom RT-DWH management platform to unify metadata, quality, permissions, and observability — reducing component count and operational complexity for small-to-medium teams.
Alibaba Cloud Hologres achieved 8,443,627 QphH@3000GB, ranking first on the TPC‑H 3TB performance list with a 54% lead over the runner‑up and the lowest cost‑per‑QphH at 390.47 CNY, demonstrating superior speed and efficiency for real‑time analytics.
This article examines the three major phases of Chinese commercial banks' core system architecture—system budding, centralized mainframe, and distributed designs—analyzes the technical improvements and shortcomings of each generation, and forecasts post‑distributed trends such as domestic‑technology adoption, cloud‑native deployment, intelligent operations, real‑time data warehousing, micro‑service structures, and large‑model integration.
Intelligent Operationsarchitecture evolutionbanking core systems
The article explains why small‑to‑medium teams need a unified web‑based management console for Flink + Paimon real‑time data warehouses, outlines the operational pain points of current script‑driven workflows, and details the proposed solution’s core modules, tech stack, and role‑based access design.
The article walks through the architecture and core workflow of a lightweight real‑time data warehouse built on Flink and Paimon, covering service‑layer refactoring, CDC task lifecycle, dual Flink submission modes, metadata synchronization, and layered security measures for small‑to‑medium enterprises.
The article examines how Xiaohongshu, Kuaishou and Meituan adopted Yunqi Lakehouse’s General Incremental Computing and Single‑Engine architecture to achieve near‑real‑time data warehouses, cutting resource usage to as low as 1/20 of full‑batch jobs, reducing data latency from days to minutes, and improving query performance.
The article analyses the structural mismatch between banks' demand for instant data‑driven decisions and their legacy, component‑heavy architectures, then details how SelectDB’s high‑performance, unified MPP engine, schema‑change agility, and financial‑grade security enable a real‑time data‑warehouse platform that supports diverse banking scenarios from dashboards to fraud detection.
The article explains how BP Claw tackles unstable AI coding results by automatically converting low‑quality PRD documents into structured, high‑quality requirements, applying token‑saving strategies, strict hallucination guards, and multi‑skill orchestration, which together boost FlinkSpec’s real‑time data‑warehouse delivery efficiency by up to 30%.
Traditional batch‑oriented data warehouses can’t keep up with AI‑driven, second‑level business needs, so the article explains what a real‑time data warehouse is, its key technical traits, business benefits such as faster decision making and cost savings, and provides a step‑by‑step implementation roadmap.
This article analyzes the architectural evolution of Alibaba Cloud Hologres from a fragmented multi‑engine data stack to the All‑in‑One design of Hologres 4.0, detailing its multimodal search, AI‑native functions, performance benchmarks, resource governance, lake integration, and real‑world deployment scenarios.
This article details a full-chain optimization of a Flink CDC real-time data warehouse pipeline syncing 20 MySQL instances to StarRocks via Kafka, resolving GC crashes, checkpoint failures, and 30-minute latency through resource scaling, parallelism redesign, memory tuning, checkpoint configuration, and startup strategy changes, achieving 5-second end-to-end latency and 100% checkpoint success.
The article explains how Hologres Dynamic Table uses a stateful incremental refresh model to efficiently handle massive historical data with tiny daily updates, dramatically reducing latency and resource consumption compared with conventional full‑refresh pipelines across several real‑world join and aggregation scenarios.
The article explains the fundamental differences between real‑time and offline data warehouses, outlines their distinct business demands, presents concrete use‑case scenarios, offers practical selection criteria, and argues that most enterprises benefit from a hybrid architecture rather than a strict either‑or choice.
data integrationdata warehouse selectionhybrid architecture
The article details Zhihu's step‑by‑step evolution from a Spark‑Streaming‑based real‑time data warehouse (v1.0) to a Flink‑Streaming SQL‑driven platform (v2.0), explaining design choices, Lambda architecture implementation, scalability challenges, and future automation plans.
This article explains how Baidu’s Feed real‑time data warehouse was rebuilt using a pure streaming architecture, detailing the limitations of the previous stream‑batch design, the technical solutions—including core/non‑core data separation, metric calculation in streaming, and Parquet storage with Apache Arrow—and the resulting cost reductions, latency improvements, and future roadmap.
This case study details how Shuhe Technology, a leading fintech company, overcame data redundancy, low resource utilization, and slow reporting by adopting Alibaba Cloud EMR Serverless StarRocks for a unified, real‑time data warehouse, achieving standardized data pipelines, cost savings, and minute‑level decision latency.
This article details Ctrip Business Travel’s implementation of a near‑real‑time data warehouse and lakehouse using Flink CDC and Apache Paimon, covering order wide‑table construction, automated ticket reminders, ad attribution, batch‑stream integration, and lessons on Partial Update, Aggregation, and Tag‑based incremental processing.
Shopee combined the Paimon data lake with StarRocks and Flink to create a quasi‑real‑time warehouse, enabling fast task diagnostics and a high‑performance financial reconciliation system while dramatically reducing storage costs and latency through innovative ODS, snapshot, and branch table techniques.
This article details Shopee's Data Infra team's use of the Paimon data lake to build near‑real‑time warehouses, accelerate ODS layers, implement a task‑diagnosis system, and create a reconciliation platform, while sharing future plans and a Q&A session.
Lalamove’s international logistics platform transformed its real‑time data warehouse by leveraging Apache Flink and the Paimon lakehouse, addressing challenges of multi‑region data centers, time‑zone diversity, frequent upstream changes, and high costs, while improving scalability, latency, and operational efficiency across global markets.
To meet Bilibili’s rapidly expanding game business, the team built a unified real-time data warehouse using Hologres and Flink that replaces the traditional Lambda stack, delivering high-throughput writes, low-latency processing, seamless offline-online integration, global deployment, and real-time support for operations, advertising, and risk analytics.
Data architecture case studyFlinkGame business data
iQIYI transformed its real‑time data warehouse by replacing a costly Kafka‑based Lambda stack with a unified stream‑batch Iceberg lake, cutting storage expenses by 90%, halving compute costs, extending data retention, and delivering minute‑level freshness for 90% of use cases while preserving second‑level processing where needed.
This article analyzes the limitations of a traditional Flink + Talos + Iceberg real‑time lakehouse, introduces Apache Paimon's lakehouse table format and LSM storage, and demonstrates three practical use cases—partial‑update widening, streaming upsert, and lookup join—showing cost, stability, and performance improvements while outlining future roadmap.
This article introduces Hologres Dynamic Table, explains its typical real‑time data‑warehouse scenarios, outlines the underlying architecture and its challenges, details the incremental and full‑refresh modes, provides usage steps, best‑practice recommendations, demo scenarios, and compares it with competing solutions.
Alibaba Mama’s advertising platform migrated from a monolithic Flink‑Kafka pipeline to a layered Paimon lakehouse, adding DWS upsert support and multi‑layer storage, which delivers minute‑level data freshness, cuts latency by 2.5 hours, reduces resource use over 40 %, halves development effort and achieves ≥99.9 % availability.
This guide explains Hologres compute group instances, covering the four core challenges of real‑time data warehouses, the architecture and benefits of elastic compute groups, time‑based elasticity, serverless computing, query queue, and step‑by‑step practices for management, authorization, connection, monitoring, and migration.
This article traces the three‑generation evolution of real‑time data warehouses—from the Lambda architecture to a lakehouse approach—detailing how Hologres, Flink, and Dynamic Table technologies enable unified storage, multi‑mode computing, serverless execution, and high‑performance analytics in modern big‑data environments.
The article presents a collection of technical abstracts from the DataFun Summit, covering XiaoHongShu's REDgraph distributed graph database, DingoDB's multimodal vector database, Tencent's Tianqiong autonomous data platform, real‑time data warehouse architectures at Douyin and Ant Group, and Alibaba Cloud's serverless ClickHouse offering, all aimed at advancing large‑scale data processing and analytics.
The article presents a collection of technical overviews—including a graph database for distributed queries, a next‑generation vector database, real‑time data warehouse architectures at Douyin and Ant Group, a cloud‑native ClickHouse service, and best practices for financial data warehousing—while also explaining how to obtain the related e‑book.
At the 2024 Data Intelligence Conference, ICBC's Big Data and AI Lab detailed the evolution of its data intelligence platform, covering architectural redesign, real‑time data warehouse technology, unified intelligent data tools, and future development directions to boost efficiency and innovation.
This article introduces Alibaba Cloud's Hologres real‑time data warehouse, explains its architecture, table and index types, and provides a step‑by‑step guide for setting up instances, importing data, and running OLAP and KV performance tests using TPC‑H benchmarks.
This article presents a comprehensive overview of the evolution from traditional Lambda‑based real‑time data warehouse solutions to a data‑lake‑integrated architecture, detailing the shortcomings of legacy designs, the iterative improvements made at JD Technology, and the technical and operational challenges encountered during implementation.
This article presents a comprehensive overview of JD Tech's real‑time data warehouse evolution, detailing the legacy Lambda‑based design, its shortcomings, the transition to a data‑lake‑integrated architecture, iterative improvements, encountered technical and non‑technical issues, and future outlooks.
This article details how CaoCao Mobility transformed its ride‑hailing platform by replacing a traditional Lambda architecture with an enterprise‑grade real‑time data warehouse built on Hologres and Flink, covering business motivations, architectural design, component capabilities, performance optimizations, operational safeguards, and future roadmap.
This article describes how China Unicom’s Western Innovation Research Institute designed and deployed a centralized, real‑time log analytics platform using Apache Doris, detailing the migration from Hive and ClickHouse, performance optimizations, storage cost reductions, and the resulting improvements in data ingestion, query speed, and operational efficiency.
Apache DorisCold‑Hot Data ManagementData Compression
This article examines the growing demand for real‑time data warehouses, compares mature streaming frameworks, evaluates Lambda, Kappa and hybrid architectures, reviews industry implementations from Didi and OPPO, and proposes a standard‑layer + stream + data‑lake solution with Apache Paimon, Hudi, and Iceberg.
This article presents a comprehensive overview of the current challenges, platform architecture, governance planning, and technical optimizations—including Flink SQL, Kafka batch processing, and partitioned stream tables—used to improve resource efficiency, stability, and scalability of a large‑scale real‑time data warehouse.
This article presents a comprehensive overview of NetEase Cloud Music's real‑time data warehouse architecture, its stream‑batch consistency model, the low‑code FastX platform implementation, and future plans for expanding lake‑house capabilities and high‑availability solutions.
This article details the three‑stage evolution of 360's real‑time data warehouse—from Storm + Druid + MySQL to Flink + Druid + TiDB and finally to Flink + Apache Doris—explaining architectural pain points, the reasons for choosing Doris, and how the new system delivers sub‑second query latency, strong consistency, and simplified operations across advertising scenarios.
ByteHouse, a cloud‑native data warehouse on Volcano Engine, leverages ClickHouse to deliver ultra‑fast real‑time analytics, detailing its business motivations, ROI‑driven evaluation, performance trade‑offs, architectural evolution, and concrete financial‑industry deployments such as real‑time monitoring and risk‑control scenarios.
ByteHouse, a cloud‑native data warehouse built on ClickHouse, delivers ultra‑fast real‑time and massive offline analytics with elastic scaling, addressing business needs in ByteDance and the financial sector through optimized architecture, ROI‑driven monitoring, and comprehensive operational tools.
This article explains how ByteDance designed and deployed a real‑time data warehouse on a data lake using Hudi, detailing three business scenarios, the challenges of latency, consistency and resource usage, and the engineering solutions—including upserts, compaction services, indexing, and future unified storage plans.
This article details how Tongcheng Data Science built a real‑time analytical data warehouse using Apache Doris, covering business scenarios, the evolution from a legacy 1.0 architecture to a Doris‑based 2.0 design, deployment topology, development workflow, operational benefits, and future roadmap.
This article details a banking‑focused real‑time data‑warehouse solution that leverages Flink Table Store to handle both incremental fact‑table updates and full‑table dimension calculations, compares three traditional approaches, and explains data ingestion, query modes, export options, and future streaming‑warehouse directions.
This article shares practical experience comparing ClickHouse and StarRocks as real‑time data warehouses, outlines the project requirements, evaluates each system's suitability for log‑type and business‑type data, and describes CDC‑based synchronization methods from MySQL to both platforms.
The article explains why enterprises need real-time data warehouses, compares Lambda and Kappa architectures, describes Ctrip Hotel's Lambda‑plus‑OLAP variant built with Flink and StarRocks, and details practical solutions for ordering, wide‑table generation, and data validation that enable billion‑row, low‑latency analytics.
This article explains why real‑time data warehouses are becoming essential, outlines their goals, compares them with traditional offline warehouses, and presents detailed design patterns, naming conventions, and case studies from Didi, Kuaishou, Tencent, Youzan and other enterprises, highlighting challenges and solutions for streaming, storage, and query layers.
Facing stricter security, accuracy, and latency demands, 360 DataTech rebuilt its real‑time data warehouse by selecting Apache Doris for its high‑performance writes, SQL compatibility, low operational complexity, and active community, then detailed the architecture, ingestion, query acceleration, monitoring, troubleshooting, and future plans.
Cross‑Express consolidated multiple big‑data engines into a unified, high‑performance analytics platform using StarRocks, achieving millisecond‑level query latency, real‑time data warehousing, significant cost savings, and improved multi‑scenario business applications; the initiative also simplified BI development, reduced hardware requirements, and set a roadmap for future engine enhancements.
This article describes Liulishuo's migration to a Flink‑based real‑time data warehouse, covering background, benefits, technology selection (storage, Flink platform, dimension table connectors), overall architecture, concrete Hudi and Elasticsearch ingestion examples, processing SQL, and future outlook for unified batch‑streaming storage.
This presentation details ByteDance's real‑time data‑warehouse implementations using Apache Hudi, covering scenario classifications, challenges of traditional offline warehouses, practical solutions for ingestion, upsert, validation, indexing, query optimization, and future plans for extensible indexing and unified batch‑stream processing.
This article reviews the evolution of data warehouse architectures, explains Amazon's serverless real-time data lake design and its key services, and details Amazon Redshift's cloud-native real-time data warehouse features, streaming ingestion, and integrated machine learning capabilities.
This article explains how Alibaba Cloud Flink and Hologres combine to deliver a one‑stop, cloud‑native real‑time data‑warehouse solution that supports low‑latency ingestion, full‑incremental CDC, automatic schema evolution, high‑performance OLAP and online serving, and simplifies ETL/ELT pipelines for enterprise analytics.
This article details the design and implementation of a real‑time data warehouse for an advertising platform, covering business background, challenges, a Lambda‑based architecture, Flink stream processing setup, ETL logic, sink handling, and performance results, concluding with future improvement directions.
This article explores the challenges of real‑time data ingestion at massive scale, explains Hologres' innovative upsert model, performance optimizations, and compares its capabilities with other data‑warehouse solutions, demonstrating how it delivers low‑latency, high‑throughput writes for big‑data analytics.
This case study details how an online education platform migrated its real‑time data warehouse from Kudu to Alibaba Cloud Hologres, overcoming technical bottlenecks, reducing operational costs by nearly a million dollars annually, and achieving higher throughput, lower latency, and easier maintenance across multiple business scenarios.
This article examines the evolution of big‑data architectures, identifies the latency and maintenance issues of classic Lambda designs, and presents a hybrid Lambda‑Kappa solution that unifies streaming and batch processing to achieve minute‑level data freshness and second‑level query latency while reducing development cost.
Data ModelingKappa architectureLambda architecture
This article presents Tencent's PCG data platform evolution, detailing the challenges of integrating multiple business groups, the design of a unified big‑data architecture, real‑time and batch processing pipelines, MQ and ATTA systems, and comprehensive operational practices for reliability and scalability.
The article uses an acorn‑moving analogy to highlight latency and traceability challenges in enterprise data warehouses, then explains offline versus real‑time approaches, compares Lambda and Kappa architectures, discusses Iceberg integration, and shares a detailed e‑commerce real‑time warehouse case study with optimization tips.
This article presents a comprehensive case study of building a real‑time advertising data warehouse at Auto Home, detailing the evaluation of streaming engines and storage solutions, the layered architecture design, implementation steps with Flink and StarRocks, monitoring practices, encountered issues, and future roadmap, demonstrating how second‑level data freshness and high accuracy were achieved.
This article details Kuaishou's real‑time data warehouse architecture, the business challenges of massive traffic and diverse requirements, and the forward‑ and reverse‑assurance strategies—including lifecycle standards, monitoring, fault‑injection testing, and a Spring Festival case study—that together ensure high stability, low latency, and sub‑0.5% accuracy for billion‑scale streaming workloads.
This article details how a fast‑growing supply‑chain platform migrated from MySQL and Hive to Apache Doris for real‑time analytics, describing the architectural evolution, the advantages of the new design, practical implementation steps, encountered challenges, and the performance and cost benefits achieved.
Zuoyebang’s data team replaced fragmented, slow query solutions with Apache Doris, building a unified real‑time data warehouse that dramatically cut query latency from hours to seconds, streamlined data modeling, and improved reliability across diverse business scenarios, while integrating with Flink, Kafka, and ES via a unified API.
This article describes a low‑code, Flink‑based real‑time data‑warehouse construction system that abstracts the warehouse building process into ODS, DWD, DWS, and ADS layers, leverages a domain‑specific language and plugin engine to reduce development effort, and details its architecture, DSL design, plugin extensibility, dimension‑table completion, stream merging, aggregation, and storage strategies.
Shopee replaced its hourly Hive pipeline with a hybrid Flink‑Hudi real‑time data warehouse that groups Kafka topics, applies lightweight stream ETL, uses partial‑update MOR tables for multi‑stream joins and COW tables for versioned batches, cutting latency from about 90 minutes to 2–30 minutes and halving resource usage.
This article explains why modern enterprises need real‑time data‑warehouse architectures, breaks down traditional layered warehouse concepts, compares Lambda and Kappa models, evaluates five practical real‑time solutions—including Iceberg‑based lakehouse and MPP databases—provides code snippets, and offers selection guidance with real‑world company examples.
This article explains the challenges of early real‑time data pipelines, introduces a layered real‑time warehouse architecture, provides step‑by‑step Flink SQL code for building a demo warehouse, and covers comprehensive data governance, quality metrics, lifecycle management, and naming conventions for production‑grade big‑data systems.
This article explores the growing demand for real‑time data warehouses, compares them with traditional offline warehouses, and presents detailed architectures, layer designs, naming conventions, and case studies from companies like Didi, Kuaishou, Tencent, and Youzan, highlighting challenges, solutions, and performance optimizations.
The article explains how Kafka serves as the core of a real‑time data warehouse for search, detailing its advantages over traditional databases, integration with Flink for low‑latency stream processing, architectural patterns such as Lambda/Kappa, scaling challenges, and comprehensive monitoring using Kafka Eagle.
Alibaba Cloud's real‑time data warehouse Hologres passed a large‑scale performance test of up to 8,192 nodes, demonstrating how cloud‑native design, automated operations, and intelligent monitoring enable ultra‑large‑scale deployment and high‑availability for enterprise data platforms.
In a festive New‑Year message, Youzan Coder’s WeChat account celebrates the Tiger Year by presenting its six‑year journey and unveiling the 2021 top‑10 most‑read technical articles—ranging from ClickHouse implementation to UI automation, TCP networking, low‑code testing, thread‑pool optimization, elite testing practices, mobile permission systems, search quality assurance, and proactive mobile monitoring—each with thousands of views, and encourages readers to keep learning in 2022.
This article explains how to build a unified stream‑batch real‑time data warehouse using FlinkSQL, covering prerequisite knowledge, five core concepts, two implementation approaches, a comparison of traditional versus real‑time architectures, and a comprehensive hands‑on example, illustrated with diagrams.
This article explains the evolution, challenges, and technical solutions for building an end‑to‑end real‑time data warehouse by combining Apache Flink's stream processing with StarRocks' ultra‑fast OLAP engine, covering architecture, data models, integration methods, best‑practice cases, and future roadmap.
This article analyzes the challenges of massive data query efficiency, explains how Flink's stream processing and ClickHouse's OLAP engine complement each other, and presents a layered real‑time data‑warehouse architecture with practical guidance on data ingestion, write strategies, quality assurance, and evolving batch‑stream integration patterns.
This tutorial explains how to use Apache Flink 1.12 to construct a unified streaming‑batch data warehouse by integrating Hive via HiveCatalog and HiveDialect, performing read/write operations, configuring upsert‑Kafka sinks, and leveraging Flink CDC connectors for change data capture from MySQL and other sources.
During the 2021 Double‑11 shopping festival, logistics provider DiSiFang upgraded its real‑time data warehouse with Flink and Hologres, enabling multi‑billion‑row joins, cutting costs by 50%, and delivering stable, low‑latency analytics that powered high‑frequency dashboards and improved overall delivery speed.
This article reviews the challenges of building a real‑time data warehouse, compares Lambda and Kappa architectures, introduces Apache Kudu’s master‑tablet design, storage model and partition strategies, and shares practical experiences and future directions for a Kudu‑based streaming analytics platform.
Meituan Waimai’s data intelligence team outlines a universal real‑time data‑warehouse methodology that combines a production platform with an interactive analytics engine, detailing scenarios, technology choices, architectural designs, platformization, SLA management, and a practical Lambda‑style case study.
This article presents Tencent's practical experience of constructing a real‑time data warehouse by integrating Apache Flink with Apache Iceberg, covering background pain points of traditional Lambda architectures, Iceberg's table format and capabilities, Flink‑Iceberg sink design, small‑file handling, and future roadmap for a unified streaming‑batch data lake.
To meet Xianyu’s billion‑event‑per‑day real‑time analysis needs, the team built a petabyte‑scale warehouse using Hologres for storage and Alibaba‑enhanced Flink (Blink) for streaming, organized into ODS, DWD, DWS, ADS and DIM layers, enabling minute‑level aggregations, rapid anomaly detection, and instant product‑team insights.
ULTRON is a one‑stop real‑time data‑warehouse development platform built on FlinkSQL that unifies data integration, asset management, cluster deployment, modeling, ETL, OLAP analysis and governance, addressing the limitations of traditional batch‑oriented warehouses and simplifying streaming data workflows for developers.
This article details Youzan's end‑to‑end experience building a real‑time data warehouse, covering background, application scenarios, layered architecture, streaming ETL techniques, data validation, recovery processes, and three production projects that illustrate the solution in action.
This article presents NetEase Game's streaming ETL solution built on Flink, covering business background, log characteristics, specialized and generic ETL services, architectural evolution, Python UDF integration, runtime optimizations, fault‑tolerance mechanisms, and future roadmap for unified real‑time and offline data warehouses.
The article analyzes the massive storage and performance challenges of a real‑time DMP cache built on Redis, outlines data characteristics and technical obstacles, and proposes eviction policies, bucket‑based hashing, and fragmentation‑reduction techniques with Java code examples to achieve billion‑scale in‑memory key‑value storage.
To meet iQIYI video production’s thousands‑QPS, petabyte‑scale, frequently‑updated data and large‑table join requirements, the team built a Spark‑plus‑ClickHouse real‑time warehouse that streams Kafka changes, joins HBase dimensions, and writes to ClickHouse, reducing reporting development time from days to hours while supporting both offline and real‑time analytics.
This article presents Tencent's experience of constructing a real‑time data warehouse by integrating Apache Flink with Apache Iceberg, covering background pain points, Iceberg's table format and capabilities, Flink‑Iceberg streaming and batch processing, practical implementations, and future roadmap for data‑lake acceleration.
This article recounts the author’s journey building a real‑time data warehouse using Flink, Kafka, Redis, and ClickHouse, describing the initial batch‑oriented setup, successive architectural evolutions, challenges with wide tables and dimension data, and the final OLAP‑centric solution with secondary caching.
This article describes how a fast‑growing company built a layered real‑time data warehouse on Flink, detailing the evolution from a simple 1.0 pipeline to a 2.0 architecture with ODS, DWD and ADS layers, dimension joins, exactly‑once sinks, HDFS partitioning, monitoring, and future improvements.
This article introduces JD's real‑time data warehouse evolution, outlines the limitations of traditional Lambda‑based warehouses, compares open‑source lake formats (Delta, Hudi, Iceberg), explains Delta Lake's transaction‑log architecture and read flow, and demonstrates how a unified batch‑stream development model simplifies data processing and improves reliability.
Alibaba’s Customer Experience (CCO) team transformed its real‑time data platform by evolving from a Lambda‑style database architecture to a cloud‑native real‑time data warehouse powered by Hologres and Flink, achieving higher throughput, lower latency, reduced costs, and self‑service analytics for massive Double‑11 traffic.
This article details Didi’s end‑to‑end construction of a real‑time data warehouse for the Ride‑Sharing (顺风车) business, covering motivations, layer‑by‑layer architecture, naming conventions, StreamSQL capabilities, operational tooling, achieved results, challenges, and future batch‑stream integration plans.