The article outlines the problems of early data services, proposes a two‑step data warehouse solution—abstracting a data model and extracting, cleaning, and storing data from MySQL, MongoDB, and Elasticsearch—then details storage selection, dimensional modeling, incremental ETL, and Airflow workflow management.
The article presents a three‑step methodology—identifying core requirements, constructing a capability‑matching matrix, and validating through pilot scenarios—to help enterprises evaluate data warehouse modeling tools based on scale, architecture compatibility, team skills, automation, extensibility, cost, performance testing, and continuous optimization.
This article details Baidu APP's massive data‑warehouse overhaul, describing the two‑step strategy that stabilized log cleaning, modernized the ETL framework, introduced wide‑table architectures, and implemented tiered storage to dramatically improve processing speed, reliability, and cost efficiency for petabyte‑scale workloads.
This article presents twelve practical Hive SQL tuning techniques—ranging from avoiding COUNT(DISTINCT) to configuring parallel execution, reducer settings, and strict mode—to help data engineers reduce data skew, eliminate small files, improve resource utilization, and significantly accelerate query execution in large‑scale data warehouse environments.
As data volumes explode, enterprises struggle with the high hardware, performance, operational, and migration costs of traditional OLAP warehouses, but ByteHouse’s cloud‑native architecture offers a cost‑effective, high‑performance solution that dramatically reduces both explicit and hidden expenses.
The article explains why scattered, heterogeneous enterprise data requires a unified, subject‑oriented data warehouse, outlines its historical evolution, compares Inmon and Kimball architectures, describes the core components and modeling stages, and details dimensional modeling techniques and best practices.
This article examines the exploding data volumes that pressure modern enterprises, outlines the explicit (hardware, performance) and implicit (operations, migration) costs of operating an OLAP‑based data warehouse, and explains how ByteHouse’s cloud‑native architecture reduces both cost categories while delivering real‑time analytics.
This article outlines a comprehensive data warehouse development framework, detailing each lifecycle phase—from requirement gathering and design to development, testing, release, and operations—while defining the responsibilities of product managers, designers, developers, testers, and ops staff to improve efficiency and reduce risk.
This article traces the three‑generation evolution of data architecture—from the structured‑data era of data warehouses, through the flexible, multi‑format data lake, to the unified lakehouse model—explaining the drivers, benefits, challenges, and future trends shaping modern data platforms.
This article outlines the fundamentals of banking data warehouses, compares them with ODS, data marts, and data lakes, and presents a detailed development workflow and management system that emphasizes standards, role division, and tool-supported automation for efficient, reliable data warehousing.
The article outlines five key ways AI large models can revamp enterprise data warehouses—automated data governance and cleaning, natural‑language query interfaces, real‑time predictive analytics, multimodal data integration with knowledge graphs, and security‑compliant automated operations—while also discussing supporting technologies, toolchains, and future trends toward industry‑specific models.
This article compiles three interview rounds for a data warehouse engineer role, covering fundamental concepts, practical skills, and leadership thinking with detailed Q&A on ETL, Hadoop components, schema design, data quality, data lake vs. warehouse, ACID properties, cloud solutions, SQL optimization, real‑time processing, security, and team management.
This article examines how artificial intelligence is reshaping data warehouses by introducing automated modeling, intelligent scheduling, SQL generation from natural language, and NLQ capabilities, while also reviewing practical tools, cloud‑native trends, and strategic steps for enterprises to adopt AI‑driven data platforms.
The article explains why a well‑designed data model is essential for data‑warehouse performance and reliability, then walks through three core modeling patterns—flat tables, master/detail, and dimensional (star) schemas—illustrating each with concrete Power BI examples, step‑by‑step design stages, and trade‑offs such as many‑to‑many relationships and bidirectional filters.
This article examines the challenges of a legacy Lambda‑based membership data warehouse, introduces Apache Paimon’s lakehouse architecture and its key features, and showcases three real‑world implementations—partial‑update order wide tables, Bitmap‑based UV counting, and branch‑based data correction—while discussing benefits, remaining challenges, and future directions.
This article explains how the Tianji advertising platform consolidated heterogeneous MySQL, ClickHouse, and TiDB data sources into a single StarRocks data warehouse, addressing data silos, real‑time performance, and query complexity while improving accuracy, latency, and development efficiency.
Facing massive data volumes, complex component dependencies, high TCO, and real‑time processing limits, the政采云 platform replaced its Hadoop stack with StarRocks’ minimalist, decoupled architecture, achieving lower costs, elastic scaling, faster queries, easier operations, and robust fault tolerance across diverse government procurement workloads.
The article outlines six typical pitfalls enterprises face when selecting a data‑warehouse modeling tool—such as chasing feature overload, ignoring business needs, overlooking team skills, and underestimating integration and long‑term costs—and provides concrete, step‑by‑step recommendations to avoid each error.
This article compiles 18 essential interview topics for data warehouse engineer roles, covering self‑introduction, architecture layering, dimensional modeling, HDFS operations, Spark vs MapReduce, join implementation, SQL challenges, OLAP selection, real‑time quality assurance, and job transition considerations.
This article compiles the complete set of interview questions used in ByteDance's data warehouse engineer hiring process, covering three interview rounds with topics ranging from self‑introduction and window functions to data skew, shuffle mechanisms, warehouse architecture, data quality, and governance, plus interviewers' feedback and recommended preparation tips.
This article chronicles ODPS's 15‑year journey from its exploratory beginnings to a modern, AI‑enabled big data platform, detailing its four development phases, architectural layers, SQL engine upgrades, real‑time processing, lakehouse integration, and the new Data+AI capabilities offered by MaxCompute and DataWorks.
This article explains the evolution from traditional data warehouses and data lakes to the unified Lakehouse architecture, detailing its design, benefits, challenges, and research directions for delivering high‑performance SQL and advanced analytics on open‑format storage.
This article details how a real‑time data warehouse built on Flink, Kafka, HBase and MySQL was redesigned using Paimon to eliminate costly deduplication, handle out‑of‑order events, enable streaming reads, simplify aggregation, replace multiple lookup sources, and achieve faster, more reliable batch repairs, resulting in major resource and operational gains.
This article details the migration of Southeast Asia's leading tech group GoTerra from Google BigQuery to Alibaba Cloud MaxCompute, explaining the performance challenges, the MaxQA accelerator architecture, optimization techniques, resource‑quota strategies, and future enhancements that together double query efficiency while reducing costs.
This article summarizes how enterprises adopt Apache Doris for lakehouse integration, near‑real‑time data warehouses, federated queries, and performance optimization, highlighting practical experiences, connector support, scheduling capabilities, and comparisons with Elasticsearch.
This article explains how to transform traditional SRE troubleshooting into a data‑driven process by pre‑collecting operational metrics into a data warehouse, using ETL to create layered data models (ODS, DIM, DWD, DWS) that enable efficient, repeatable analysis while balancing data freshness and storage costs.
An interviewee recounts a challenging first interview that focused on Flink resource configuration, late data handling, and offline data warehouse design, and shares practical advice on attitude, thorough preparation, emphasizing real project storytelling, and post‑interview review to continuously improve performance.
This guide walks you through every essential topic—from data warehouse architecture and layering, through ETL, OLAP, Hadoop, and Flink, to visualization tools, learning paths, recommended resources, and the management skills needed to become a proficient data warehouse architect.
The article examines a real-world migration of a major Southeast Asian tech group from Google BigQuery to Alibaba Cloud MaxCompute, highlighting the challenges of complex data types, the columnar storage and execution engine redesigns, and the resulting performance gains that often surpass BigQuery.
This article presents ten practical Hive SQL advanced scenarios—including session segmentation, funnel conversion, median calculation, array explosion, hierarchical recursion, deduplication, small‑file merging, conditional aggregation, approximate statistics, and data‑quality checks—each with full SQL code, key technical points, and optimization tips for Hive 3.x.
This article details the complex migration of a leading Southeast Asian tech group's data warehouse from Google BigQuery to Alibaba Cloud MaxCompute, outlining challenges such as storage format differences, SQL compatibility, and performance tuning, and explains how the new Append DeltaTable format with dynamic bucketing and incremental reclustering resolves these issues.
This article explains how Baidu’s search data team redesigned its data warehouse with wide‑table modeling, Parquet columnar storage, and a Spark‑ClickHouse fusion engine, eliminating redundancy, cutting query latency from minutes to seconds, and enabling self‑service analytics for thousands of users.
This article outlines Baidu’s innovative approach to building its search data platform, detailing the design of wide‑table models, the upgrade to a Spark‑based fusion computation engine, and the new Turing 3.0 service delivery framework, which together deliver higher efficiency, lower cost, and faster, more reliable analytics.
Suixingfu rebuilt its payment data pipeline by replacing a fragmented Lambda stack with a unified Porter CDC + StarRocks + Elasticsearch architecture, achieving three‑fold query speed, ten‑fold analytics efficiency, 20% storage reduction, and sub‑second data‑capture latency across high‑concurrency, ad‑hoc, and batch workloads.
After a weekend dive into Apache Hive’s official Wiki and GitHub, this article highlights Hive’s declining visibility compared to Spark and Flink, examines its 4.0 release’s major features—including Iceberg integration, enhanced ACID, cost‑based optimizer upgrades, and Ozone support—while reflecting on its role in modern data ecosystems.
This article examines how AI large models reshape enterprise data warehouses through intelligent data governance, natural‑language query conversion, real‑time predictive analytics, multimodal knowledge integration, and automated security compliance, while outlining supporting technologies, toolchains, and future trends.
This article walks through Apache Doris’s lakehouse‑in‑one architecture, explains its core value and paradigm, details the system’s components and use cases, examines technical challenges such as file‑format diversity and I/O stability, and presents a suite of optimizations—from predicate push‑down and partition pruning to metadata caching and dynamic scheduling—that dramatically improve query performance and resource utilization, while also outlining future roadmap plans.
This article explains the core components of a data warehouse ecosystem, distinguishes fact and dimension tables, outlines synchronization strategies, introduces star, snowflake, and constellation schemas, and details the layered architecture from ODS to data marts for effective big‑data analytics.
This article details the evolution of a fast‑growing finance reporting system from a monolithic microservice architecture plagued by data inconsistency, low efficiency, and scalability limits to a robust, high‑performance big‑data warehouse built with layered data models, SparkSQL processing, and unified scheduling, highlighting design decisions, technical trade‑offs, and measurable performance gains.
The article shares practical experiences using large language models such as Cursor and DeepSeek in data‑warehouse workflows, covering assisted coding, automated metric extraction, self‑service analysis, documentation generation, their benefits, limitations, and the broader impact on data engineering roles.
This article walks through using MySQL to pivot rows into columns, covering storage concepts, practical CASE‑WHEN aggregation techniques, and detailed SQL examples for sum, count, average and max calculations, while also discussing performance considerations for medium‑scale and large‑scale data workloads.
This article details the evolution of a fast‑growing e‑commerce finance platform from a monolithic microservice architecture plagued by data inconsistency, low processing efficiency, and scalability limits to a robust, distributed big‑data warehouse using SparkSQL, layered data models, and optimized scheduling, achieving ten‑fold performance gains and near‑zero failure rates.
This article reviews common interview questions for data development roles, covering Spark stage partitioning and optimization, criteria for evaluating data warehouses, Flink's handling of late data, and provides practical answers and resources to help candidates deliver standout responses.
This article explains data architecture, its purpose, historical development, major enterprise frameworks, various data management system types, and the key advantages it brings to organizations, helping readers understand how to design and implement effective data solutions in modern environments.
This article presents ten practical Hive SQL scenarios—including window functions for ranking, LAG for time‑interval analysis, random‑salt techniques to mitigate data skew, dynamic partition writes, JSON parsing with UDFs, retention calculations, consecutive‑login detection, regex‑based path analysis, CUBE multi‑dimensional aggregation, and ORC storage optimizations—each accompanied by optimization tips and complete code examples.
Facing a flood of data from over 8,000 communities, the Bifeng service team migrated from a monolithic storage‑compute architecture to a StarRocks‑based storage‑compute separation solution, achieving lower costs, higher resource utilization, faster queries, and improved SLA across their property management platform.
DeepSeek leverages AI to automate multi‑source integration, data cleaning, warehouse modeling, real‑time processing, governance, metadata management, reporting, cloud scaling, and decision support, offering twelve distinct use cases that boost efficiency, intelligence, and scalability of modern data warehouses.
This article compiles a comprehensive set of Spark interview questions frequently asked by leading tech companies, providing detailed explanations of Spark’s performance mechanisms, architecture, RDD persistence, checkpointing, streaming, dependency types, HA setup, and practical coding examples to help data warehouse engineers prepare effectively.
This article, based on a meetup presentation, details Alibaba Cloud's MaxCompute platform—its evolution, serverless architecture, AI integration, distributed Python framework, Object Table, near‑real‑time processing, and intelligent warehouse features—addressing the challenges of data warehouses in the Data+AI era.
DeepSeek leverages large‑model AI to automate requirement analysis, intelligent modeling, performance tuning, and value extraction in data warehouses, addressing low development efficiency, high O&M cost, latency, and lack of intelligence while showcasing concrete use‑case results across finance, e‑commerce, and manufacturing.
This article details the business background, technical challenges, and the step‑by‑step implementation of an intelligent data warehouse on Alibaba Cloud MaxCompute, covering offline data pipelines, metric calculation, data analysis, and future plans for data lake and AI‑driven analytics.
The article explores nine fundamental questions about data‑warehouse modeling—why and when to model, how to evaluate and compare models, the warehouse’s unique role versus business systems, modern architectural shifts, a quantitative value‑proof scoring framework, industry‑standard versus custom approaches, demonstrating business impact, and career insights—concluding that true value lies in enabling informed decisions rather than technology hype.
This document outlines a comprehensive data warehouse model design and review process, covering revision records, project overview, business requirements, conceptual and logical modeling, ETL workflow, exception handling, and acceptance criteria with practical examples and templates.
This article introduces MaxFrame, a cloud‑native distributed Python compute service built on MaxCompute, detailing its architecture, seamless integration with the Python ecosystem, and real‑world use cases ranging from large‑scale data analysis and machine learning to offline LLM inference and custom image deployments.
This article explains the concept of Slowly Changing Dimensions (SCD) in data warehouses, compares six common SCD handling methods—including SCD0, SCD1, SCD2, SCD3, combined SCD2+SCD3, and historical tables—and guides you on selecting the most suitable approach for your business needs.
This article explains the role of the DWS layer in data warehouses, outlines design principles, step‑by‑step modeling, naming conventions, field design, provides concrete DDL/ETL examples, common pitfalls, and how to build reusable, performant summary tables for analytics.
This article provides a comprehensive guide to designing the Data Warehouse Detail (DWD) layer, covering Kimball‑based design principles, step‑by‑step modeling, table and field naming conventions, concrete Hive DDL/DML examples, and optimization techniques such as partitioning, bucketing, and compression.
This article explains the role of dimension tables in data warehouses, compares natural and surrogate keys, and details three strategies—overwrite, insert new rows, and add new attributes—for managing slowly changing dimensions, including full‑snapshot tables.
This article explains MaxCompute’s intelligent data warehouse architecture, its self‑learning optimization pipeline, the role of intelligent materialized views, the automated recommendation system for materialized views, and the AutoMV feature that automatically creates, updates, and cleans up materialized views to reduce compute costs and improve query performance.
This article analyzes the challenges of Alibaba Mama's brand advertising data warehouse built on a Lambda architecture, introduces Apache Paimon lake storage and Dolphin OLAP engine as a unified lakehouse solution, details implementation steps, performance gains, and business benefits across multiple advertising scenarios.
Xiaohongshu tackled low reuse of APP tables, limited scalability of single-table BI datasets, and poor dashboard query performance by introducing logical datasets and materialized views, which enable query pruning, reduce data redundancy, and accelerate BI queries, achieving up to 80% latency reduction and higher hit rates.
DeepSeek, the popular Chinese large‑language model, boosts data‑warehouse engineers' productivity by offering free, open‑source AI assistance across code writing, model design, metadata management, data quality monitoring, and governance, ultimately maximizing enterprise data asset value.
This article summarizes a meetup talk by Alibaba Cloud expert Yu Deshui, detailing MaxCompute’s evolution, serverless architecture, AI‑enabled features, and the platform’s comprehensive solutions—including OpenLake, MaxFrame, Object Table, near‑real‑time computing, and AI Functions—to address the challenges of modern data‑centric AI workloads.
The article explains the three detailed-granularity fact table types—transaction, periodic snapshot, and cumulative snapshot—detailing their purposes, design principles, and comparative usage, and offers a simplified interpretation to help data engineers choose the appropriate fact table for data warehouse modeling.
The article walks through the DIKW model, illustrates how a homestay platform’s data stack evolves from a simple MVP to a complex, AI‑enabled distributed system, and details the design of a unified Data Warebase that combines database and data‑warehouse capabilities to meet performance, correctness, and real‑time demands, backed by concrete case studies and measurable improvements.
This article presents the motivation, design principles, architecture, metric system, and results of a data‑warehouse evaluation framework that quantifies efficiency, quality, cost, and model health to drive systematic governance and continuous improvement across the organization.
This article explains how to construct a comprehensive data metric system—including classification of atomic and derived indicators, top‑down and bottom‑up design methods, data‑warehouse layer architecture, and BI analysis types—to ensure consistent, actionable insights across the enterprise.
The article outlines Didi’s end‑to‑end metric lifecycle—from background, requirements and current pain points to a multi‑stage solution that introduces a unified metric dictionary, management tool, logical modeling, and consumption layer—to achieve accurate, timely, consistent, and efficiently managed indicators across the data warehouse ecosystem.
This guide introduces StarRocks, a high‑performance MPP database, explains its architecture and typical use cases, walks through a Docker‑based quickstart, shows how to create databases and tables, load NYC crash and weather datasets via Stream Load, and demonstrates analytical SQL queries that reveal traffic‑accident patterns under different weather conditions.
This article explains how Baidu’s TDE‑ClickHouse, as a core engine of the Turing 3.0 ecosystem, overcomes platform fragmentation, quality issues, and usability challenges through the OneData+ development paradigm, multi‑level aggregation, projection, query‑caching, bulk‑load ingestion, and a cloud‑native architecture to achieve sub‑second query response for massive data volumes.
This article outlines the motivations, objectives, and architectural design of a next‑generation data management platform, detailing its four‑layer “four‑ization” approach, core services such as data integration, modeling, API provisioning, componentization, as well as governance, security, and operational best practices.
The article outlines a step‑by‑step methodology for creating Retrieval‑Augmented Generation and custom Agent applications with LangChain, illustrated by an AI assistant for activity components that evolves from a rapid Dify prototype to a LangChain‑based RAG system and finally a hand‑crafted ReAct‑style agent, detailing LCEL chain composition, vector‑search integration, model performance trade‑offs, and a unified routing layer.
Baidu MEG’s TDE‑ClickHouse optimization in the Turing 3.0 ecosystem boosts query speed up to 10×, halves latency, enables billion‑row bulk imports in under two hours, and migrates to a cloud‑native, ZooKeeper‑free architecture supporting 350 k CPU cores, 10 PB storage, and sub‑3‑second responses for 150 k daily BI queries.
Apache Doris 3.0 introduces storage‑compute separation, native lakehouse write‑back, optimized Variant handling for semi‑structured data, stronger ETL transaction support, enhanced multi‑table materialized views, and Java UDTF capabilities, providing developers with more flexible, cost‑effective, and high‑performance analytics solutions.
This article explains DataFusion, a Rust‑written, Arrow‑based query engine that offers high performance, extensibility, and seamless integration with various data sources, detailing its architecture, execution model, Rust advantages, and practical usage examples for building modern data‑warehouse solutions.
The paper presents Baidu’s Mobile Operations Promotion Platform redesign, introducing an online‑offline integrated task‑scheduling architecture that partitions settlement fields to the data‑warehouse, records all jobs in a unified MySQL operation table, orchestrates them via Turing Data Studio, and manages dependencies to achieve consistent, auditable, billion‑scale settlement processing.
This article outlines Didi's approach to metric management, covering background, data product overview, and challenges in traditional and agile BI models, and presents a comprehensive solution for metric standardization, logical modeling, quality assurance, unified consumption, and future roadmap to improve data warehouse efficiency and consistency.
This article compiles concise English summaries of several technical papers covering Xiaohongshu's REDgraph graph database, DingoDB vector database, Tianqiong autonomous data platform, Douyin's real‑time data warehouse, financial‑grade data warehousing, Alibaba Cloud ClickHouse Serverless offering, best practices in financial data governance, and 58.com user‑profile data warehouse construction.
This article presents a collection of technical abstracts covering graph database parallel query optimization, next‑generation vector databases, real‑time data warehouse architectures, and cloud‑native analytics solutions, while also providing instructions for obtaining the full e‑book via a WeChat public account.
The DataFun Summit page presents a collection of technical papers covering graph database parallel queries, next‑generation vector databases, real‑time data warehouse architectures, and best practices in finance and e‑commerce, while also providing instructions for obtaining the e‑book via a public account.
Qimao, a Shanghai‑based cultural entertainment internet firm, details its 20 PB big‑data architecture built on StarRocks, Flink, Hive, and Redis, covering data ingestion, real‑time processing, audience selection, metric anomaly drill‑down, 730‑day aggregation, and future plans for metric acceleration and full‑link data governance.
This article examines the common benefits, drawbacks, and design principles of layered architecture across micro‑service, data‑warehouse, and protocol designs, illustrating each point with real‑world examples and offering practical guidance on when and how to apply layering effectively.
Layered architecture offers abstract stability, functional reuse, cohesion, hidden complexity, and scalability, but can introduce extra complexity, performance overhead, and dependency risk, so designers should retain essential layers, enforce one‑way cross‑layer calls, depend only on lower layers, keep lower layers stable, and ensure each layer has a clear purpose.
This article presents Xide International's cross‑border e‑commerce data platform, detailing its multi‑layer business architecture, the scalability and data‑access problems encountered, and how a Protonbase‑driven data‑warehouse and micro‑service redesign dramatically improved query speed, operational efficiency, and cost.
This article walks through the fundamentals of data modeling, explaining what data is, the DIKW framework, why modeling matters, and detailing the end‑to‑end process from conceptual design through logical and physical layers, including DIM, DWD, DWS, and ADM tables with practical tips and naming conventions.
The article details Kuaishou's adoption of a data lake powered by Apache Hudi, covering the challenges of growing data warehouses, the migration from Hive to Hudi, concrete business case studies, promotion strategies, and key takeaways for large‑scale data engineering.
This article presents Kuaishou's comprehensive white‑box cost governance practice over the past year, detailing the data‑governance framework, engine and data‑warehouse white‑boxing techniques, compression algorithm replacement, HBO automatic tuning, operator analysis, and the resulting performance and cost benefits, as well as future plans.
This article explains how Hive serves as a data‑warehouse layer for user‑profile tagging, covering data‑warehouse fundamentals, fact‑and‑dimension modeling, partitioned storage, label aggregation, and ID‑mapping techniques with practical Hive DDL/DML examples.
This article presents Kuaishou's experience of designing and implementing a Doris‑driven lakehouse integrated analytics system, covering the current OLAP landscape, challenges of data duplication and governance, the new architecture with caching and auto‑materialization, implementation details, performance impact, and future work.
To achieve sub‑second, multi‑dimensional analytics on Bilibili’s hundred‑million‑row datasets, the team built a ClickHouse‑based acceleration layer that combines grouping‑set pre‑aggregation, bitmap (RBM) distinct handling, and a dense dictionary encoding service, dramatically cutting CPU, memory and query latency versus traditional OLAP pipelines.
This article outlines practical approaches to data warehouse construction, covering dimensional modeling, layered architecture, capability development, real‑time and batch processing with technologies like Hive, Spark, Flink, Iceberg, and discusses governance, security, and future trends toward data value and real‑time metrics.
This article explores how PostgreSQL’s advanced features—UNLOGGED tables, JSONB, SKIP LOCKED, TimescaleDB, pg_cron, PostGIS, full‑text search, JSON generation, pgaudit, and GraphQL adapters—can replace specialized tools like Kafka, Redis, MongoDB, and others, simplifying the tech stack while boosting performance and maintainability.
This article presents Qichacha's comprehensive data‑cost‑reduction strategy, detailing its Hadoop‑based three‑pillar architecture, layered data warehouse, Hive upgrades, unified metadata across multi‑cloud clusters, middleware choices such as Alluxio and JuiceFS, version‑compatible hybrid clouds, and Kubernetes‑driven resource orchestration to achieve scalable, low‑cost data processing.
This article presents a comprehensive overview of Xiaomi's sales data warehouse, detailing its development history, dimensional modeling theory, multi‑layer architecture, Lambda design with batch and streaming processing, capability layers, security measures, and answers to common technical questions.
This article presents TCHouse‑C, a cloud‑native ClickHouse service, detailing its real‑time data update architecture, schema‑less ingestion, various update strategies such as Delete‑Insert and lightweight‑update/delete, and comprehensive performance tests comparing UniqueMergeTree with standard ClickHouse engines across import, query, and update workloads.
This article explains how to construct a full‑chain data lineage system, covering its overall architecture, quality measurement framework, and application layer, and demonstrates practical use cases such as handling data growth, monitoring warehouse changes, accelerating development, ensuring consistency, and automating metric decomposition in real‑world business scenarios.
This article outlines the step‑by‑step evolution of Baidu's Feed data warehouse—from traditional layered modeling to hour‑level core tables, then real‑time wide tables, and finally a flow‑batch integrated multi‑version wide‑table architecture—highlighting the motivations, design choices, challenges, and resulting benefits.
This article explains how to construct a financial indicator system and its underlying data warehouse using FineBI, covering metric hierarchy design, ODS/DWD/DWS/ADS layers, AI‑enhanced BI analysis, and practical tips for organizational adoption and governance.
This article details JD Retail's transition from a complex Lambda architecture to a unified real‑time data pipeline using Flink, Hudi, and StarRocks, addressing data completeness versus latency, reducing maintenance costs, improving storage efficiency, and delivering faster, more consistent analytics for business users.
This article details Kuashou's journey of adopting the Hudi data lake, covering business challenges, migration from Hive to Hudi, architectural redesign, promotion strategy, real‑world use cases such as CDC sync and batch‑stream integration, and key lessons learned for large‑scale data engineering.
