BestHub
Sign in
Tagged articles

data warehouse

644 articles · Page 2 of 7
Data Integration and Governance
Data Integration and Governance
Dec 4, 2025 · Big Data

Why Data Warehouses Fail? 4 Essential Building Blocks Explained

The article examines common data‑warehouse pitfalls—poor data quality, endless ad‑hoc queries, and unreadable SQL—then outlines a systematic approach that includes correcting mindset, designing a layered architecture, applying disciplined modeling practices, enforcing data governance, and managing engineering details for sustainable success.

Data ModelingETLdata warehouse
0 likes · 12 min read
Why Data Warehouses Fail? 4 Essential Building Blocks Explained
Smart Sea Tide
Smart Sea Tide
Nov 28, 2025 · Big Data

A Comprehensive Panorama of Data Modeling for Big Data Systems

The article explains why structured data modeling is essential for large‑scale systems, compares OLTP and OLAP approaches, details ER, dimensional, Data Vault and Anchor methodologies, and walks through Alibaba's multi‑stage data‑modeling practice, including OneData implementation, dimension design, fact‑table types, and aggregation strategies.

AlibabaData ModelingOneData
0 likes · 54 min read
A Comprehensive Panorama of Data Modeling for Big Data Systems
Big Data Tech Team
Big Data Tech Team
Nov 24, 2025 · Big Data

Avoid the 5 Common DWS Design Traps and Build Scalable Data Warehouses

This article analyzes the five typical pitfalls when designing DWS aggregation tables—from chimney‑style schemas to performance blind spots—explains their consequences, and provides concrete, production‑ready recommendations, code examples, and design principles to create reusable, efficient data‑warehouse layers.

AggregationDWS DesignPerformance Optimization
0 likes · 10 min read
Avoid the 5 Common DWS Design Traps and Build Scalable Data Warehouses
Alibaba Cloud Big Data AI Platform
Alibaba Cloud Big Data AI Platform
Nov 19, 2025 · Big Data

How We Migrated 100k BigQuery SQL Scripts to MaxCompute Using AST and LLM Automation

This article details a real‑world migration of a Southeast Asian tech group’s data warehouse from Google BigQuery to Alibaba Cloud MaxCompute, describing the challenges of converting 100,000 SQL scripts, the AST‑driven and LLM‑assisted automation pipeline, rule‑engine iteration, quality control, and the measurable performance and cost benefits achieved.

ASTBigQueryLLM
0 likes · 12 min read
How We Migrated 100k BigQuery SQL Scripts to MaxCompute Using AST and LLM Automation
Smart Sea Tide
Smart Sea Tide
Nov 4, 2025 · Big Data

Implementing an Integrated Data Lake and Lakehouse Architecture with Apache Iceberg and Flink

The article explains the concepts of data lakes and lakehouses, compares them with traditional data warehouses, outlines the reliability, performance, and security challenges of data lakes, and then details a practical lakehouse implementation using Apache Iceberg, Flink SQL, CDC pipelines, and supporting tools such as Hive Metastore and Trino.

Apache IcebergCDCData Lake
0 likes · 18 min read
Implementing an Integrated Data Lake and Lakehouse Architecture with Apache Iceberg and Flink
Big Data Tech Team
Big Data Tech Team
Nov 2, 2025 · Big Data

Data Governance Blueprint: Naming Rules, Lifecycle Levels, and Layered Architecture

Explore a comprehensive data governance guide covering naming conventions, data lifecycle classifications, layered architecture standards, inter-layer calling rules, and model design principles, providing practical standards and best practices for building robust, maintainable data warehouses and analytics platforms.

Data LifecycleModel Designdata warehouse
0 likes · 9 min read
Data Governance Blueprint: Naming Rules, Lifecycle Levels, and Layered Architecture
Smart Sea Tide
Smart Sea Tide
Oct 31, 2025 · Big Data

Practical Guide to Building a Data Warehouse Metric System

The article analyzes common pain points in data‑warehouse metric management from business, technical, and product perspectives, outlines unified goals, defines a comprehensive model—including data domains, dimensions, atomic, derived, and composite metrics—and presents a step‑by‑step process for designing, implementing, and governing a scalable metric system.

Business AnalyticsDWM layerData Modeling
0 likes · 13 min read
Practical Guide to Building a Data Warehouse Metric System
Big Data Tech Team
Big Data Tech Team
Oct 28, 2025 · Big Data

From Data Chaos to Decision Engine: A Step‑by‑Step Guide to Offline Data Warehouse Governance

This article walks you through why unmanaged data warehouses fail, outlines three golden governance principles, details five practical implementation steps—from building a data lineage map to creating business‑driven quality dashboards—and shares real‑world case studies and common pitfalls to help turn your data warehouse into a trusted decision‑making engine.

Business IntelligenceData Qualitydata warehouse
0 likes · 11 min read
From Data Chaos to Decision Engine: A Step‑by‑Step Guide to Offline Data Warehouse Governance
Big Data Tech Team
Big Data Tech Team
Oct 26, 2025 · Big Data

Data Domain vs Subject Area: Clear Differences and Practical Guide

This article explains the distinct concepts of data domain and subject area, uses a library‑vs‑bookstore analogy, presents a real e‑commerce case, compares them in a concise table, and offers best‑practice steps and common pitfalls to help data teams design efficient data architectures.

Data DomainData ModelingSubject Area
0 likes · 8 min read
Data Domain vs Subject Area: Clear Differences and Practical Guide
Smart Sea Tide
Smart Sea Tide
Oct 17, 2025 · Big Data

Building a Data Warehouse for the Data Processing Layer

The article outlines the problems of early data services, proposes a two‑step data warehouse solution—abstracting a data model and extracting, cleaning, and storing data from MySQL, MongoDB, and Elasticsearch—then details storage selection, dimensional modeling, incremental ETL, and Airflow workflow management.

AirflowETLMySQL
0 likes · 12 min read
Building a Data Warehouse for the Data Processing Layer
Smart Sea Tide
Smart Sea Tide
Oct 14, 2025 · Industry Insights

How to Choose the Right Data Warehouse Modeling Tool for Your Enterprise

The article presents a three‑step methodology—identifying core requirements, constructing a capability‑matching matrix, and validating through pilot scenarios—to help enterprises evaluate data warehouse modeling tools based on scale, architecture compatibility, team skills, automation, extensibility, cost, performance testing, and continuous optimization.

Enterprise Datadata warehouseevaluation matrix
0 likes · 7 min read
How to Choose the Right Data Warehouse Modeling Tool for Your Enterprise
Baidu Geek Talk
Baidu Geek Talk
Oct 13, 2025 · Big Data

How Baidu Scaled Its Data Warehouse to Handle Billions of PVs and Petabytes

This article details Baidu APP's massive data‑warehouse overhaul, describing the two‑step strategy that stabilized log cleaning, modernized the ETL framework, introduced wide‑table architectures, and implemented tiered storage to dramatically improve processing speed, reliability, and cost efficiency for petabyte‑scale workloads.

ETLPerformance Optimizationbig data
0 likes · 25 min read
How Baidu Scaled Its Data Warehouse to Handle Billions of PVs and Petabytes
Big Data Tech Team
Big Data Tech Team
Oct 10, 2025 · Big Data

12 Essential Hive SQL Optimization Tricks to Boost Query Performance

This article presents twelve practical Hive SQL tuning techniques—ranging from avoiding COUNT(DISTINCT) to configuring parallel execution, reducer settings, and strict mode—to help data engineers reduce data skew, eliminate small files, improve resource utilization, and significantly accelerate query execution in large‑scale data warehouse environments.

HiveSQL optimizationdata warehouse
0 likes · 11 min read
12 Essential Hive SQL Optimization Tricks to Boost Query Performance
DataFunTalk
DataFunTalk
Oct 8, 2025 · Big Data

How ByteHouse Cuts Data Warehouse Costs: Tackling Explicit and Implicit Challenges

As data volumes explode, enterprises struggle with the high hardware, performance, operational, and migration costs of traditional OLAP warehouses, but ByteHouse’s cloud‑native architecture offers a cost‑effective, high‑performance solution that dramatically reduces both explicit and hidden expenses.

ByteHouseOLAPbig data
0 likes · 6 min read
How ByteHouse Cuts Data Warehouse Costs: Tackling Explicit and Implicit Challenges
Smart Sea Tide
Smart Sea Tide
Sep 29, 2025 · Databases

Data Warehouse Architecture and Construction Methodology

The article explains why scattered, heterogeneous enterprise data requires a unified, subject‑oriented data warehouse, outlines its historical evolution, compares Inmon and Kimball architectures, describes the core components and modeling stages, and details dimensional modeling techniques and best practices.

Business IntelligenceData IntegrationETL
0 likes · 48 min read
Data Warehouse Architecture and Construction Methodology
DataFunSummit
DataFunSummit
Sep 28, 2025 · Big Data

How ByteHouse Cuts Data Warehouse Costs: Tackling Hidden and Visible Expenses

This article examines the exploding data volumes that pressure modern enterprises, outlines the explicit (hardware, performance) and implicit (operations, migration) costs of operating an OLAP‑based data warehouse, and explains how ByteHouse’s cloud‑native architecture reduces both cost categories while delivering real‑time analytics.

ByteHouseOLAPbig data
0 likes · 5 min read
How ByteHouse Cuts Data Warehouse Costs: Tackling Hidden and Visible Expenses
Smart Sea Tide
Smart Sea Tide
Sep 22, 2025 · Big Data

Data Warehouse Development Standards: Phase Planning, Roles, and End-to-End Process

This article outlines a comprehensive data warehouse development framework, detailing each lifecycle phase—from requirement gathering and design to development, testing, release, and operations—while defining the responsibilities of product managers, designers, developers, testers, and ops staff to improve efficiency and reduce risk.

DesignETLdata warehouse
0 likes · 16 min read
Data Warehouse Development Standards: Phase Planning, Roles, and End-to-End Process
IT Architects Alliance
IT Architects Alliance
Sep 21, 2025 · Big Data

From Data Warehouses to Lakehouses: Why Data Architecture Keeps Evolving

This article traces the three‑generation evolution of data architecture—from the structured‑data era of data warehouses, through the flexible, multi‑format data lake, to the unified lakehouse model—explaining the drivers, benefits, challenges, and future trends shaping modern data platforms.

Data LakeLakehousedata architecture
0 likes · 11 min read
From Data Warehouses to Lakehouses: Why Data Architecture Keeps Evolving
Big Data Tech Team
Big Data Tech Team
Sep 16, 2025 · Industry Insights

How AI Large Models Transform Enterprise Data Warehouses

The article outlines five key ways AI large models can revamp enterprise data warehouses—automated data governance and cleaning, natural‑language query interfaces, real‑time predictive analytics, multimodal data integration with knowledge graphs, and security‑compliant automated operations—while also discussing supporting technologies, toolchains, and future trends toward industry‑specific models.

AIdata warehouseenterprise analytics
0 likes · 7 min read
How AI Large Models Transform Enterprise Data Warehouses
Big Data Tech Team
Big Data Tech Team
Sep 15, 2025 · Interview Experience

Top Data Warehouse Engineer Interview Questions & Answers Revealed

This article compiles three interview rounds for a data warehouse engineer role, covering fundamental concepts, practical skills, and leadership thinking with detailed Q&A on ETL, Hadoop components, schema design, data quality, data lake vs. warehouse, ACID properties, cloud solutions, SQL optimization, real‑time processing, security, and team management.

ETLHadoopSQL optimization
0 likes · 12 min read
Top Data Warehouse Engineer Interview Questions & Answers Revealed
Big Data Tech Team
Big Data Tech Team
Sep 14, 2025 · Industry Insights

How AI Is Transforming Data Warehouses: Automation, SQL Generation, and NLQ

This article examines how artificial intelligence is reshaping data warehouses by introducing automated modeling, intelligent scheduling, SQL generation from natural language, and NLQ capabilities, while also reviewing practical tools, cloud‑native trends, and strategic steps for enterprises to adopt AI‑driven data platforms.

AINLQautomation
0 likes · 8 min read
How AI Is Transforming Data Warehouses: Automation, SQL Generation, and NLQ
Smart Sea Tide
Smart Sea Tide
Sep 12, 2025 · Fundamentals

Practical Data Modeling Patterns for Data Warehousing

The article explains why a well‑designed data model is essential for data‑warehouse performance and reliability, then walks through three core modeling patterns—flat tables, master/detail, and dimensional (star) schemas—illustrating each with concrete Power BI examples, step‑by‑step design stages, and trade‑offs such as many‑to‑many relationships and bidirectional filters.

Data ModelingPower BIdata warehouse
0 likes · 11 min read
Practical Data Modeling Patterns for Data Warehousing
360 Zhihui Cloud Developer
360 Zhihui Cloud Developer
Sep 11, 2025 · Big Data

How Paimon Transforms Membership Data Warehousing: From Legacy Lambda to Real‑Time Lakehouse

This article examines the challenges of a legacy Lambda‑based membership data warehouse, introduces Apache Paimon’s lakehouse architecture and its key features, and showcases three real‑world implementations—partial‑update order wide tables, Bitmap‑based UV counting, and branch‑based data correction—while discussing benefits, remaining challenges, and future directions.

Data LakeFlinkPaimon
0 likes · 29 min read
How Paimon Transforms Membership Data Warehousing: From Legacy Lambda to Real‑Time Lakehouse
StarRocks
StarRocks
Sep 9, 2025 · Big Data

From Hadoop to StarRocks: Revamping a Government Procurement Data Platform

Facing massive data volumes, complex component dependencies, high TCO, and real‑time processing limits, the政采云 platform replaced its Hadoop stack with StarRocks’ minimalist, decoupled architecture, achieving lower costs, elastic scaling, faster queries, easier operations, and robust fault tolerance across diverse government procurement workloads.

Hadoop migrationStarRockscloud-native
0 likes · 16 min read
From Hadoop to StarRocks: Revamping a Government Procurement Data Platform
Smart Sea Tide
Smart Sea Tide
Sep 9, 2025 · Industry Insights

6 Common Mistakes in Choosing an Enterprise Data Warehouse Modeling Tool and How to Avoid Them

The article outlines six typical pitfalls enterprises face when selecting a data‑warehouse modeling tool—such as chasing feature overload, ignoring business needs, overlooking team skills, and underestimating integration and long‑term costs—and provides concrete, step‑by‑step recommendations to avoid each error.

avoidance strategydata warehouseenterprise
0 likes · 11 min read
6 Common Mistakes in Choosing an Enterprise Data Warehouse Modeling Tool and How to Avoid Them
Big Data Tech Team
Big Data Tech Team
Aug 21, 2025 · Big Data

Crack ByteDance Data Warehouse Engineer Interviews: 30+ Essential Questions

This article compiles the complete set of interview questions used in ByteDance's data warehouse engineer hiring process, covering three interview rounds with topics ranging from self‑introduction and window functions to data skew, shuffle mechanisms, warehouse architecture, data quality, and governance, plus interviewers' feedback and recommended preparation tips.

ByteDanceSQLdata warehouse
0 likes · 6 min read
Crack ByteDance Data Warehouse Engineer Interviews: 30+ Essential Questions
Alibaba Cloud Big Data AI Platform
Alibaba Cloud Big Data AI Platform
Aug 13, 2025 · Big Data

How ODPS Evolved Over 15 Years into a Next‑Gen AI‑Ready Big Data Platform

This article chronicles ODPS's 15‑year journey from its exploratory beginnings to a modern, AI‑enabled big data platform, detailing its four development phases, architectural layers, SQL engine upgrades, real‑time processing, lakehouse integration, and the new Data+AI capabilities offered by MaxCompute and DataWorks.

AI IntegrationDataWorksMaxCompute
0 likes · 12 min read
How ODPS Evolved Over 15 Years into a Next‑Gen AI‑Ready Big Data Platform
Big Data Technology Tribe
Big Data Technology Tribe
Aug 12, 2025 · Databases

Why Lakehouse Architecture Is Redefining Modern Data Platforms

This article explains the evolution from traditional data warehouses and data lakes to the unified Lakehouse architecture, detailing its design, benefits, challenges, and research directions for delivering high‑performance SQL and advanced analytics on open‑format storage.

Data LakeLakehouseMetadata Layer
0 likes · 20 min read
Why Lakehouse Architecture Is Redefining Modern Data Platforms
58 Tech
58 Tech
Aug 7, 2025 · Big Data

Transform Real‑Time Data Warehousing with Paimon: From Flink ROW_NUMBER to Streaming Lakehouse

This article details how a real‑time data warehouse built on Flink, Kafka, HBase and MySQL was redesigned using Paimon to eliminate costly deduplication, handle out‑of‑order events, enable streaming reads, simplify aggregation, replace multiple lookup sources, and achieve faster, more reliable batch repairs, resulting in major resource and operational gains.

FlinkLakehousePaimon
0 likes · 24 min read
Transform Real‑Time Data Warehousing with Paimon: From Flink ROW_NUMBER to Streaming Lakehouse
Alibaba Cloud Big Data AI Platform
Alibaba Cloud Big Data AI Platform
Aug 5, 2025 · Big Data

How MaxQA Supercharges Query Performance for Large‑Scale Data Warehouses

This article details the migration of Southeast Asia's leading tech group GoTerra from Google BigQuery to Alibaba Cloud MaxCompute, explaining the performance challenges, the MaxQA accelerator architecture, optimization techniques, resource‑quota strategies, and future enhancements that together double query efficiency while reducing costs.

Performance OptimizationSQLbig data
0 likes · 19 min read
How MaxQA Supercharges Query Performance for Large‑Scale Data Warehouses
Big Data Technology & Architecture
Big Data Technology & Architecture
Jul 29, 2025 · Big Data

What Interviewers Really Ask About Flink, Data Consistency, and Warehouse Design

An interviewee recounts a challenging first interview that focused on Flink resource configuration, late data handling, and offline data warehouse design, and shares practical advice on attitude, thorough preparation, emphasizing real project storytelling, and post‑interview review to continuously improve performance.

Data ConsistencyFlinkInterview Preparation
0 likes · 4 min read
What Interviewers Really Ask About Flink, Data Consistency, and Warehouse Design
Big Data Tech Team
Big Data Tech Team
Jul 23, 2025 · Big Data

From Beginner to Data Warehouse Architect: A Complete Roadmap

This guide walks you through every essential topic—from data warehouse architecture and layering, through ETL, OLAP, Hadoop, and Flink, to visualization tools, learning paths, recommended resources, and the management skills needed to become a proficient data warehouse architect.

ETLFlinkHadoop
0 likes · 9 min read
From Beginner to Data Warehouse Architect: A Complete Roadmap
Big Data Tech Team
Big Data Tech Team
Jul 17, 2025 · Big Data

Master Hive SQL: 10 Advanced Use Cases & Performance Optimizations for Hive 3.x

This article presents ten practical Hive SQL advanced scenarios—including session segmentation, funnel conversion, median calculation, array explosion, hierarchical recursion, deduplication, small‑file merging, conditional aggregation, approximate statistics, and data‑quality checks—each with full SQL code, key technical points, and optimization tips for Hive 3.x.

HiveOptimizationSQL
0 likes · 9 min read
Master Hive SQL: 10 Advanced Use Cases & Performance Optimizations for Hive 3.x
Alibaba Cloud Big Data AI Platform
Alibaba Cloud Big Data AI Platform
Jul 15, 2025 · Big Data

How MaxCompute’s Append DeltaTable Transforms BigQuery Migration

This article details the complex migration of a leading Southeast Asian tech group's data warehouse from Google BigQuery to Alibaba Cloud MaxCompute, outlining challenges such as storage format differences, SQL compatibility, and performance tuning, and explains how the new Append DeltaTable format with dynamic bucketing and incremental reclustering resolves these issues.

big datadata migrationdata warehouse
0 likes · 19 min read
How MaxCompute’s Append DeltaTable Transforms BigQuery Migration
Architect
Architect
Jul 7, 2025 · Big Data

How Baidu’s New Search Data Warehouse Architecture Boosts Performance by 5×

This article explains how Baidu’s search data team redesigned its data warehouse with wide‑table modeling, Parquet columnar storage, and a Spark‑ClickHouse fusion engine, eliminating redundancy, cutting query latency from minutes to seconds, and enabling self‑service analytics for thousands of users.

Data ModelingETLParquet
0 likes · 21 min read
How Baidu’s New Search Data Warehouse Architecture Boosts Performance by 5×
Baidu Geek Talk
Baidu Geek Talk
Jul 2, 2025 · Big Data

Baidu’s Secret to Faster Search Data: Wide‑Table Modeling & Fusion Engine

This article outlines Baidu’s innovative approach to building its search data platform, detailing the design of wide‑table models, the upgrade to a Spark‑based fusion computation engine, and the new Turing 3.0 service delivery framework, which together deliver higher efficiency, lower cost, and faster, more reliable analytics.

Fusion EngineSearch AnalyticsTuring 3.0
0 likes · 21 min read
Baidu’s Secret to Faster Search Data: Wide‑Table Modeling & Fusion Engine
StarRocks
StarRocks
Jul 1, 2025 · Big Data

How StarRocks Boosted Suixingfu’s Real‑Time Data Platform: 3× Faster Queries & 10× Faster Analytics

Suixingfu rebuilt its payment data pipeline by replacing a fragmented Lambda stack with a unified Porter CDC + StarRocks + Elasticsearch architecture, achieving three‑fold query speed, ten‑fold analytics efficiency, 20% storage reduction, and sub‑second data‑capture latency across high‑concurrency, ad‑hoc, and batch workloads.

CDCFlinkReal-time Analytics
0 likes · 14 min read
How StarRocks Boosted Suixingfu’s Real‑Time Data Platform: 3× Faster Queries & 10× Faster Analytics
Big Data Technology & Architecture
Big Data Technology & Architecture
Jul 1, 2025 · Big Data

What’s New in Apache Hive 4.0? Key Features and Industry Outlook

After a weekend dive into Apache Hive’s official Wiki and GitHub, this article highlights Hive’s declining visibility compared to Spark and Flink, examines its 4.0 release’s major features—including Iceberg integration, enhanced ACID, cost‑based optimizer upgrades, and Ozone support—while reflecting on its role in modern data ecosystems.

Apache HiveHive 4.0Ozone
0 likes · 4 min read
What’s New in Apache Hive 4.0? Key Features and Industry Outlook
Big Data Tech Team
Big Data Tech Team
Jun 24, 2025 · Artificial Intelligence

How AI Large Models Are Revolutionizing Enterprise Data Warehouses

This article examines how AI large models reshape enterprise data warehouses through intelligent data governance, natural‑language query conversion, real‑time predictive analytics, multimodal knowledge integration, and automated security compliance, while outlining supporting technologies, toolchains, and future trends.

AIdata warehouselarge models
0 likes · 7 min read
How AI Large Models Are Revolutionizing Enterprise Data Warehouses
DataFunSummit
DataFunSummit
Jun 22, 2025 · Databases

Unlocking Apache Doris: How Lakehouse Integration Supercharges Data Analytics

This article walks through Apache Doris’s lakehouse‑in‑one architecture, explains its core value and paradigm, details the system’s components and use cases, examines technical challenges such as file‑format diversity and I/O stability, and presents a suite of optimizations—from predicate push‑down and partition pruning to metadata caching and dynamic scheduling—that dramatically improve query performance and resource utilization, while also outlining future roadmap plans.

Apache DorisLakehousePerformance Optimization
0 likes · 22 min read
Unlocking Apache Doris: How Lakehouse Integration Supercharges Data Analytics
Architect's Guide
Architect's Guide
Jun 14, 2025 · Big Data

Mastering Data Warehouse Design: From Fact Tables to Dimensional Modeling

This article explains the core components of a data warehouse ecosystem, distinguishes fact and dimension tables, outlines synchronization strategies, introduces star, snowflake, and constellation schemas, and details the layered architecture from ODS to data marts for effective big‑data analytics.

ETLFact Tabledata warehouse
0 likes · 15 min read
Mastering Data Warehouse Design: From Fact Tables to Dimensional Modeling
Sohu Tech Products
Sohu Tech Products
Jun 11, 2025 · Big Data

How We Transformed a Microservice Finance System into a Scalable Big Data Warehouse

This article details the evolution of a fast‑growing finance reporting system from a monolithic microservice architecture plagued by data inconsistency, low efficiency, and scalability limits to a robust, high‑performance big‑data warehouse built with layered data models, SparkSQL processing, and unified scheduling, highlighting design decisions, technical trade‑offs, and measurable performance gains.

Spark SQLarchitecture evolutiondata warehouse
0 likes · 23 min read
How We Transformed a Microservice Finance System into a Scalable Big Data Warehouse
dbaplus Community
dbaplus Community
Jun 7, 2025 · Artificial Intelligence

How Large Language Models Are Transforming Data Warehousing: Real-World Experiments and Lessons

The article shares practical experiences using large language models such as Cursor and DeepSeek in data‑warehouse workflows, covering assisted coding, automated metric extraction, self‑service analysis, documentation generation, their benefits, limitations, and the broader impact on data engineering roles.

AI automationBusiness IntelligenceLLM
0 likes · 9 min read
How Large Language Models Are Transforming Data Warehousing: Real-World Experiments and Lessons
Lin is Dream
Lin is Dream
May 25, 2025 · Databases

Master MySQL Row-to-Column Pivot: From Basics to Advanced Aggregations

This article walks through using MySQL to pivot rows into columns, covering storage concepts, practical CASE‑WHEN aggregation techniques, and detailed SQL examples for sum, count, average and max calculations, while also discussing performance considerations for medium‑scale and large‑scale data workloads.

AggregationMySQLOLAP
0 likes · 7 min read
Master MySQL Row-to-Column Pivot: From Basics to Advanced Aggregations
Zhuanzhuan Tech
Zhuanzhuan Tech
May 21, 2025 · Big Data

How We Turned a Microservice Finance System into a Scalable Big‑Data Warehouse

This article details the evolution of a fast‑growing e‑commerce finance platform from a monolithic microservice architecture plagued by data inconsistency, low processing efficiency, and scalability limits to a robust, distributed big‑data warehouse using SparkSQL, layered data models, and optimized scheduling, achieving ten‑fold performance gains and near‑zero failure rates.

ETLPerformance OptimizationSpark SQL
0 likes · 21 min read
How We Turned a Microservice Finance System into a Scalable Big‑Data Warehouse
Big Data Tech Team
Big Data Tech Team
Apr 27, 2025 · Big Data

10 Advanced Hive SQL Use Cases: Windows, Skew, JSON, and More

This article presents ten practical Hive SQL scenarios—including window functions for ranking, LAG for time‑interval analysis, random‑salt techniques to mitigate data skew, dynamic partition writes, JSON parsing with UDFs, retention calculations, consecutive‑login detection, regex‑based path analysis, CUBE multi‑dimensional aggregation, and ORC storage optimizations—each accompanied by optimization tips and complete code examples.

HivePerformance OptimizationSQL
0 likes · 9 min read
10 Advanced Hive SQL Use Cases: Windows, Skew, JSON, and More
Alibaba Cloud Big Data AI Platform
Alibaba Cloud Big Data AI Platform
Apr 27, 2025 · Big Data

Scaling Property Services: StarRocks‑Powered Storage‑Compute Separation for 8000+ Communities

Facing a flood of data from over 8,000 communities, the Bifeng service team migrated from a monolithic storage‑compute architecture to a StarRocks‑based storage‑compute separation solution, achieving lower costs, higher resource utilization, faster queries, and improved SLA across their property management platform.

Infrastructure MigrationOLAPStarRocks
0 likes · 11 min read
Scaling Property Services: StarRocks‑Powered Storage‑Compute Separation for 8000+ Communities
Big Data Tech Team
Big Data Tech Team
Apr 23, 2025 · Industry Insights

10 Powerful Ways DeepSeek Transforms Data Warehousing

DeepSeek leverages AI to automate multi‑source integration, data cleaning, warehouse modeling, real‑time processing, governance, metadata management, reporting, cloud scaling, and decision support, offering twelve distinct use cases that boost efficiency, intelligence, and scalability of modern data warehouses.

AIDeepSeekdata warehouse
0 likes · 9 min read
10 Powerful Ways DeepSeek Transforms Data Warehousing
Big Data Tech Team
Big Data Tech Team
Apr 17, 2025 · Big Data

Essential Spark Interview Q&A: Master Data Warehouse Engineer Questions

This article compiles a comprehensive set of Spark interview questions frequently asked by leading tech companies, providing detailed explanations of Spark’s performance mechanisms, architecture, RDD persistence, checkpointing, streaming, dependency types, HA setup, and practical coding examples to help data warehouse engineers prepare effectively.

RDDSparkSpark Streaming
0 likes · 21 min read
Essential Spark Interview Q&A: Master Data Warehouse Engineer Questions
Big Data Technology & Architecture
Big Data Technology & Architecture
Apr 17, 2025 · Big Data

MaxCompute: Intelligent Data Warehouse Platform for the Data+AI Era

This article, based on a meetup presentation, details Alibaba Cloud's MaxCompute platform—its evolution, serverless architecture, AI integration, distributed Python framework, Object Table, near‑real‑time processing, and intelligent warehouse features—addressing the challenges of data warehouses in the Data+AI era.

Distributed ComputingMaxComputeObject Table
0 likes · 11 min read
MaxCompute: Intelligent Data Warehouse Platform for the Data+AI Era
Big Data Tech Team
Big Data Tech Team
Apr 14, 2025 · Industry Insights

How DeepSeek AI is Transforming Data Warehouses: From Automation to Real‑Time Insights

DeepSeek leverages large‑model AI to automate requirement analysis, intelligent modeling, performance tuning, and value extraction in data warehouses, addressing low development efficiency, high O&M cost, latency, and lack of intelligence while showcasing concrete use‑case results across finance, e‑commerce, and manufacturing.

AIDeepSeekautomation
0 likes · 9 min read
How DeepSeek AI is Transforming Data Warehouses: From Automation to Real‑Time Insights
Didi Tech
Didi Tech
Mar 20, 2025 · Big Data

Key Questions and Value Assessment in Data Warehouse Modeling and Development

The article explores nine fundamental questions about data‑warehouse modeling—why and when to model, how to evaluate and compare models, the warehouse’s unique role versus business systems, modern architectural shifts, a quantitative value‑proof scoring framework, industry‑standard versus custom approaches, demonstrating business impact, and career insights—concluding that true value lies in enabling informed decisions rather than technology hype.

AIData Modelingbig data
0 likes · 12 min read
Key Questions and Value Assessment in Data Warehouse Modeling and Development
Big Data Tech Team
Big Data Tech Team
Mar 17, 2025 · Big Data

How to Design and Review a Data Warehouse Model: A Complete Guide

This document outlines a comprehensive data warehouse model design and review process, covering revision records, project overview, business requirements, conceptual and logical modeling, ETL workflow, exception handling, and acceptance criteria with practical examples and templates.

Data ModelingETLModel Design
0 likes · 6 min read
How to Design and Review a Data Warehouse Model: A Complete Guide
Alibaba Cloud Big Data AI Platform
Alibaba Cloud Big Data AI Platform
Mar 17, 2025 · Big Data

How MaxFrame Enables Scalable Python AI Workloads on MaxCompute

This article introduces MaxFrame, a cloud‑native distributed Python compute service built on MaxCompute, detailing its architecture, seamless integration with the Python ecosystem, and real‑world use cases ranging from large‑scale data analysis and machine learning to offline LLM inference and custom image deployments.

Distributed ComputingMaxFramePython
0 likes · 18 min read
How MaxFrame Enables Scalable Python AI Workloads on MaxCompute
Ma Wei Says
Ma Wei Says
Mar 11, 2025 · Big Data

Mastering DWS Layer Design: Principles, Steps, and Best Practices

This article explains the role of the DWS layer in data warehouses, outlines design principles, step‑by‑step modeling, naming conventions, field design, provides concrete DDL/ETL examples, common pitfalls, and how to build reusable, performant summary tables for analytics.

DWS LayerETLSQL
0 likes · 15 min read
Mastering DWS Layer Design: Principles, Steps, and Best Practices
Ma Wei Says
Ma Wei Says
Mar 9, 2025 · Big Data

Mastering DWD Layer Design: Principles, Fact Tables, and Performance Tips

This article provides a comprehensive guide to designing the Data Warehouse Detail (DWD) layer, covering Kimball‑based design principles, step‑by‑step modeling, table and field naming conventions, concrete Hive DDL/DML examples, and optimization techniques such as partitioning, bucketing, and compression.

DWDData ModelingFact Table
0 likes · 21 min read
Mastering DWD Layer Design: Principles, Fact Tables, and Performance Tips
Alibaba Cloud Big Data AI Platform
Alibaba Cloud Big Data AI Platform
Feb 28, 2025 · Databases

How MaxCompute’s Intelligent Data Warehouse Optimizes Queries with AutoMV

This article explains MaxCompute’s intelligent data warehouse architecture, its self‑learning optimization pipeline, the role of intelligent materialized views, the automated recommendation system for materialized views, and the AutoMV feature that automatically creates, updates, and cleans up materialized views to reduce compute costs and improve query performance.

AutoMVMaxComputeQuery Optimization
0 likes · 17 min read
How MaxCompute’s Intelligent Data Warehouse Optimizes Queries with AutoMV
Alimama Tech
Alimama Tech
Feb 21, 2025 · Industry Insights

How Paimon + Dolphin Transform Alibaba’s Brand Data Warehouse for Real‑Time Insights

This article analyzes the challenges of Alibaba Mama's brand advertising data warehouse built on a Lambda architecture, introduces Apache Paimon lake storage and Dolphin OLAP engine as a unified lakehouse solution, details implementation steps, performance gains, and business benefits across multiple advertising scenarios.

DolphinLakehouseOLAP
0 likes · 15 min read
How Paimon + Dolphin Transform Alibaba’s Brand Data Warehouse for Real‑Time Insights
Xiaohongshu Tech REDtech
Xiaohongshu Tech REDtech
Feb 20, 2025 · Big Data

How Xiaohongshu Accelerated Data Warehouse Queries with Logical Datasets & Materialized Views

Xiaohongshu tackled low reuse of APP tables, limited scalability of single-table BI datasets, and poor dashboard query performance by introducing logical datasets and materialized views, which enable query pruning, reduce data redundancy, and accelerate BI queries, achieving up to 80% latency reduction and higher hit rates.

BIMaterialized ViewQuery Optimization
0 likes · 25 min read
How Xiaohongshu Accelerated Data Warehouse Queries with Logical Datasets & Materialized Views
Big Data Tech Team
Big Data Tech Team
Feb 17, 2025 · Industry Insights

How DeepSeek Transforms Data Warehouse Development: 5 Game-Changing Benefits

DeepSeek, the popular Chinese large‑language model, boosts data‑warehouse engineers' productivity by offering free, open‑source AI assistance across code writing, model design, metadata management, data quality monitoring, and governance, ultimately maximizing enterprise data asset value.

Artificial IntelligenceData QualityDeepSeek
0 likes · 5 min read
How DeepSeek Transforms Data Warehouse Development: 5 Game-Changing Benefits
Alibaba Cloud Big Data AI Platform
Alibaba Cloud Big Data AI Platform
Feb 14, 2025 · Big Data

How MaxCompute Powers Intelligent Data Warehousing in the Data+AI Era

This article summarizes a meetup talk by Alibaba Cloud expert Yu Deshui, detailing MaxCompute’s evolution, serverless architecture, AI‑enabled features, and the platform’s comprehensive solutions—including OpenLake, MaxFrame, Object Table, near‑real‑time computing, and AI Functions—to address the challenges of modern data‑centric AI workloads.

AI IntegrationDistributed ComputingMaxCompute
0 likes · 13 min read
How MaxCompute Powers Intelligent Data Warehousing in the Data+AI Era
Big Data Technology & Architecture
Big Data Technology & Architecture
Dec 26, 2024 · Fundamentals

Detailed Granularity Fact Tables (DWD): Types, Design Principles, and Comparison

The article explains the three detailed-granularity fact table types—transaction, periodic snapshot, and cumulative snapshot—detailing their purposes, design principles, and comparative usage, and offers a simplified interpretation to help data engineers choose the appropriate fact table for data warehouse modeling.

DWDData ModelingFact Table
0 likes · 5 min read
Detailed Granularity Fact Tables (DWD): Types, Design Principles, and Comparison
Smart Era Software Development
Smart Era Software Development
Dec 25, 2024 · Databases

From Zero to One: Building a Next‑Gen Distributed Data Architecture for the AI Era

The article walks through the DIKW model, illustrates how a homestay platform’s data stack evolves from a simple MVP to a complex, AI‑enabled distributed system, and details the design of a unified Data Warebase that combines database and data‑warehouse capabilities to meet performance, correctness, and real‑time demands, backed by concrete case studies and measurable improvements.

AI IntegrationDistributed Architecturebig data
0 likes · 31 min read
From Zero to One: Building a Next‑Gen Distributed Data Architecture for the AI Era
Data Thinking Notes
Data Thinking Notes
Dec 3, 2024 · Fundamentals

How to Build a Robust Data Metric System: From Design to BI Application

This article explains how to construct a comprehensive data metric system—including classification of atomic and derived indicators, top‑down and bottom‑up design methods, data‑warehouse layer architecture, and BI analysis types—to ensure consistent, actionable insights across the enterprise.

BI analysisBusiness IntelligenceData Metrics
0 likes · 11 min read
How to Build a Robust Data Metric System: From Design to BI Application
DataFunSummit
DataFunSummit
Nov 29, 2024 · Big Data

Standardizing Metric Management in Didi’s Data Platform

The article outlines Didi’s end‑to‑end metric lifecycle—from background, requirements and current pain points to a multi‑stage solution that introduces a unified metric dictionary, management tool, logical modeling, and consumption layer—to achieve accurate, timely, consistent, and efficiently managed indicators across the data warehouse ecosystem.

Data ModelingStandardizationbig data
0 likes · 20 min read
Standardizing Metric Management in Didi’s Data Platform
Open Source Tech Hub
Open Source Tech Hub
Nov 16, 2024 · Databases

Build Real‑Time Analytics with StarRocks: Quickstart Tutorial and Sample Queries

This guide introduces StarRocks, a high‑performance MPP database, explains its architecture and typical use cases, walks through a Docker‑based quickstart, shows how to create databases and tables, load NYC crash and weather datasets via Stream Load, and demonstrates analytical SQL queries that reveal traffic‑accident patterns under different weather conditions.

MPP databaseReal-time AnalyticsSQL
0 likes · 18 min read
Build Real‑Time Analytics with StarRocks: Quickstart Tutorial and Sample Queries
Architecture & Thinking
Architecture & Thinking
Nov 15, 2024 · Databases

How Baidu’s TDE‑ClickHouse Delivers Sub‑Second Analytics on Billion‑Row Datasets

This article explains how Baidu’s TDE‑ClickHouse, as a core engine of the Turing 3.0 ecosystem, overcomes platform fragmentation, quality issues, and usability challenges through the OneData+ development paradigm, multi‑level aggregation, projection, query‑caching, bulk‑load ingestion, and a cloud‑native architecture to achieve sub‑second query response for massive data volumes.

ClickHousePerformance Optimizationbig data
0 likes · 22 min read
How Baidu’s TDE‑ClickHouse Delivers Sub‑Second Analytics on Billion‑Row Datasets
Data Thinking Notes
Data Thinking Notes
Nov 5, 2024 · Big Data

How a Next‑Gen Data Management Platform Boosts Efficiency and Innovation

This article outlines the motivations, objectives, and architectural design of a next‑generation data management platform, detailing its four‑layer “four‑ization” approach, core services such as data integration, modeling, API provisioning, componentization, as well as governance, security, and operational best practices.

Data Integrationbig datadata governance
0 likes · 20 min read
How a Next‑Gen Data Management Platform Boosts Efficiency and Innovation
37 Interactive Technology Team
37 Interactive Technology Team
Nov 4, 2024 · Artificial Intelligence

Developing RAG and Agent Applications with LangChain: A Case Study of an AI Assistant for Activity Components

The article outlines a step‑by‑step methodology for creating Retrieval‑Augmented Generation and custom Agent applications with LangChain, illustrated by an AI assistant for activity components that evolves from a rapid Dify prototype to a LangChain‑based RAG system and finally a hand‑crafted ReAct‑style agent, detailing LCEL chain composition, vector‑search integration, model performance trade‑offs, and a unified routing layer.

AI assistantAgentCloud-native
0 likes · 6 min read
Developing RAG and Agent Applications with LangChain: A Case Study of an AI Assistant for Activity Components
Baidu Geek Talk
Baidu Geek Talk
Oct 21, 2024 · Databases

TDE-ClickHouse Optimization Practice at Baidu MEG: Query Performance, Data Import, and Distributed Architecture

Baidu MEG’s TDE‑ClickHouse optimization in the Turing 3.0 ecosystem boosts query speed up to 10×, halves latency, enables billion‑row bulk imports in under two hours, and migrates to a cloud‑native, ZooKeeper‑free architecture supporting 350 k CPU cores, 10 PB storage, and sub‑3‑second responses for 150 k daily BI queries.

Baidu MEGClickHousebulkload
0 likes · 19 min read
TDE-ClickHouse Optimization Practice at Baidu MEG: Query Performance, Data Import, and Distributed Architecture
Big Data Technology & Architecture
Big Data Technology & Architecture
Oct 21, 2024 · Big Data

Key New Features of Apache Doris 3.0: Storage‑Compute Separation, Lakehouse Integration, Semi‑Structured Data, ETL Enhancements, Materialized Views, and Java UDTF

Apache Doris 3.0 introduces storage‑compute separation, native lakehouse write‑back, optimized Variant handling for semi‑structured data, stronger ETL transaction support, enhanced multi‑table materialized views, and Java UDTF capabilities, providing developers with more flexible, cost‑effective, and high‑performance analytics solutions.

Apache DorisETLJava UDTF
0 likes · 7 min read
Key New Features of Apache Doris 3.0: Storage‑Compute Separation, Lakehouse Integration, Semi‑Structured Data, ETL Enhancements, Materialized Views, and Java UDTF
360 Tech Engineering
360 Tech Engineering
Oct 17, 2024 · Databases

Introducing DataFusion: A High‑Performance Rust‑Based Query Engine Powered by Apache Arrow

This article explains DataFusion, a Rust‑written, Arrow‑based query engine that offers high performance, extensibility, and seamless integration with various data sources, detailing its architecture, execution model, Rust advantages, and practical usage examples for building modern data‑warehouse solutions.

Apache ArrowDataFusionQuery Engine
0 likes · 15 min read
Introducing DataFusion: A High‑Performance Rust‑Based Query Engine Powered by Apache Arrow
Baidu Tech Salon
Baidu Tech Salon
Oct 16, 2024 · Big Data

Design and Implementation of an Online/Offline Integrated Task Scheduling System for Baidu's Mobile Operations Promotion Platform (OPS)

The paper presents Baidu’s Mobile Operations Promotion Platform redesign, introducing an online‑offline integrated task‑scheduling architecture that partitions settlement fields to the data‑warehouse, records all jobs in a unified MySQL operation table, orchestrates them via Turing Data Studio, and manages dependencies to achieve consistent, auditable, billion‑scale settlement processing.

Baidudata warehouseoffline processing
0 likes · 14 min read
Design and Implementation of an Online/Offline Integrated Task Scheduling System for Baidu's Mobile Operations Promotion Platform (OPS)
DataFunTalk
DataFunTalk
Sep 28, 2024 · Big Data

Metric Management and Standardization in Didi's Data Platform

This article outlines Didi's approach to metric management, covering background, data product overview, and challenges in traditional and agile BI models, and presents a comprehensive solution for metric standardization, logical modeling, quality assurance, unified consumption, and future roadmap to improve data warehouse efficiency and consistency.

BIData ModelingStandardization
0 likes · 21 min read
Metric Management and Standardization in Didi's Data Platform
DataFunTalk
DataFunTalk
Sep 20, 2024 · Databases

Technical Paper Summaries on Graph Databases, Vector Databases, and Real-Time Data Warehousing

This article compiles concise English summaries of several technical papers covering Xiaohongshu's REDgraph graph database, DingoDB vector database, Tianqiong autonomous data platform, Douyin's real‑time data warehouse, financial‑grade data warehousing, Alibaba Cloud ClickHouse Serverless offering, best practices in financial data governance, and 58.com user‑profile data warehouse construction.

big datadata warehousegraph database
0 likes · 5 min read
Technical Paper Summaries on Graph Databases, Vector Databases, and Real-Time Data Warehousing
DataFunTalk
DataFunTalk
Sep 17, 2024 · Databases

Overview of Recent Advances in Graph, Vector, and Real-Time Data Warehouse Technologies

This article presents a collection of technical abstracts covering graph database parallel query optimization, next‑generation vector databases, real‑time data warehouse architectures, and cloud‑native analytics solutions, while also providing instructions for obtaining the full e‑book via a WeChat public account.

big datacloud-nativedata warehouse
0 likes · 5 min read
Overview of Recent Advances in Graph, Vector, and Real-Time Data Warehouse Technologies
DataFunSummit
DataFunSummit
Sep 16, 2024 · Databases

DataFun Summit: Technical Papers on Graph Databases, Vector Databases, Real‑Time Data Warehouses and Industry Data Practices

The DataFun Summit page presents a collection of technical papers covering graph database parallel queries, next‑generation vector databases, real‑time data warehouse architectures, and best practices in finance and e‑commerce, while also providing instructions for obtaining the e‑book via a public account.

Real-time Analyticsbig datadata warehouse
0 likes · 5 min read
DataFun Summit: Technical Papers on Graph Databases, Vector Databases, Real‑Time Data Warehouses and Industry Data Practices
Alibaba Cloud Big Data AI Platform
Alibaba Cloud Big Data AI Platform
Sep 13, 2024 · Big Data

How Qimao Scales 20PB Data with StarRocks, Flink, and Real‑Time Analytics

Qimao, a Shanghai‑based cultural entertainment internet firm, details its 20 PB big‑data architecture built on StarRocks, Flink, Hive, and Redis, covering data ingestion, real‑time processing, audience selection, metric anomaly drill‑down, 730‑day aggregation, and future plans for metric acceleration and full‑link data governance.

FlinkStarRocksbig data
0 likes · 13 min read
How Qimao Scales 20PB Data with StarRocks, Flink, and Real‑Time Analytics
DevOps
DevOps
Sep 12, 2024 · Fundamentals

Advantages, Disadvantages, and Principles of Layered Architecture

This article examines the common benefits, drawbacks, and design principles of layered architecture across micro‑service, data‑warehouse, and protocol designs, illustrating each point with real‑world examples and offering practical guidance on when and how to apply layering effectively.

data warehousedesign principleslayered architecture
0 likes · 11 min read
Advantages, Disadvantages, and Principles of Layered Architecture
Tencent Cloud Developer
Tencent Cloud Developer
Sep 11, 2024 · Fundamentals

Advantages, Disadvantages, and Principles of Layered Architecture in Software Systems

Layered architecture offers abstract stability, functional reuse, cohesion, hidden complexity, and scalability, but can introduce extra complexity, performance overhead, and dependency risk, so designers should retain essential layers, enforce one‑way cross‑layer calls, depend only on lower layers, keep lower layers stable, and ensure each layer has a clear purpose.

DDDdata warehousemicroservices
0 likes · 11 min read
Advantages, Disadvantages, and Principles of Layered Architecture in Software Systems
DataFunSummit
DataFunSummit
Sep 8, 2024 · Big Data

Building and Optimizing a Cross‑Border E‑Commerce Data Platform: Architecture, Challenges, and Protonbase‑Based Solutions

This article presents Xide International's cross‑border e‑commerce data platform, detailing its multi‑layer business architecture, the scalability and data‑access problems encountered, and how a Protonbase‑driven data‑warehouse and micro‑service redesign dramatically improved query speed, operational efficiency, and cost.

Cross-border E-commercebig datadata platform
0 likes · 11 min read
Building and Optimizing a Cross‑Border E‑Commerce Data Platform: Architecture, Challenges, and Protonbase‑Based Solutions
Alibaba Cloud Developer
Alibaba Cloud Developer
Sep 3, 2024 · Big Data

Mastering Data Modeling: From Raw Data to Insightful Warehouses

This article walks through the fundamentals of data modeling, explaining what data is, the DIKW framework, why modeling matters, and detailing the end‑to‑end process from conceptual design through logical and physical layers, including DIM, DWD, DWS, and ADM tables with practical tips and naming conventions.

Data ModelingETLdata warehouse
0 likes · 11 min read
Mastering Data Modeling: From Raw Data to Insightful Warehouses
DataFunTalk
DataFunTalk
Aug 27, 2024 · Big Data

Kuaishou's Year-Long White‑Box Cost Governance in Big Data: Engine, Data‑Warehouse, and Tool Optimizations

This article presents Kuaishou's comprehensive white‑box cost governance practice over the past year, detailing the data‑governance framework, engine and data‑warehouse white‑boxing techniques, compression algorithm replacement, HBO automatic tuning, operator analysis, and the resulting performance and cost benefits, as well as future plans.

big datacost optimizationdata warehouse
0 likes · 29 min read
Kuaishou's Year-Long White‑Box Cost Governance in Big Data: Engine, Data‑Warehouse, and Tool Optimizations